Papers with Large language models
Logic Haystacks: Probing LLMs’ Long-Context Logical Reasoning (Without Easily Identifiable Unrelated Padding) (2026.eacl-short)
Copied to clipboard
| Challenge: | Recent large language models claim long context windows, but evaluations often involve simple retrieval tasks or synthetic tasks padded with irrelevant text. |
| Approach: | They use grammars to generate simplified English with logical representations to create long input text while controlling its semantics. |
| Outcome: | The proposed model performs better with realistic distractors than with standard models. |
How LLMs React to Industrial Spatio-Temporal Data? Assessing Hallucination with a Novel Traffic Incident Benchmark Dataset (2025.naacl-industry)
Copied to clipboard
Qiang Li, Mingkun Tan, Xun Zhao, Dan Zhang, Daoan Zhang, Shengzhao Lei, Anderson S. Chu, Lujun Li, Porawit Kamnoedboon
| Challenge: | Large language models (LLMs) have a high potential to digitize and enhance the health & public services industry. |
| Approach: | They propose to use a cross-lingual benchmark dataset to assess the robustness of state-of-the-art LLMs in the spatio vs temporal domain for traffic incident classification. |
| Outcome: | The proposed model performs well in the spatio-temporal domain and in the non-English context. |
Rethinking the Evaluation of Alignment Methods: Insights into Diversity, Generalisation, and Safety (2026.eacl-srw)
Copied to clipboard
Denis Janiak, Julia Moska, Dawid Motyka, Karolina Seweryn, Paweł Walkowiak, Bartosz Żuk, Arkadiusz Janz
| Challenge: | Existing studies focus on individual techniques or specific dimensions, lacking a holistic assessment of the inherent trade-offs. |
| Approach: | They propose a framework that compares LLM alignment methods across five axes . they use a validated LLM-as-judge prompt to compare the results . |
| Outcome: | The proposed framework compares LLM alignment methods across factuality, safety, conciseness, proactivity, diversity and safety axes . it provides insights into trade-offs of common alignment methods, guiding the development of more balanced and reliable LLMs. |
Benchmarking LLM’s Capability in Reasoning over Conflicting Web References (2026.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) integrated with retrieval-augmented generation (RAG) are a dominant framework for building intelligent assistants. |
| Approach: | They propose a benchmark to evaluate LLMs' reasoning capability over real-world conflicting documents retrieved from the web. |
| Outcome: | The proposed benchmark evaluates LLMs' reasoning capability over real-world conflicting documents retrieved from the web. |
Constructing a Dataset for Hallucination Detection in Japanese Summarization with Fine-grained Faithfulness Labels (2026.eacl-srw)
Copied to clipboard
| Challenge: | Large language models (LLMs) can generate fluent text, but the quality of generated content depends on its consistency with the given input. |
| Approach: | They constructed a Japanese evaluation dataset for hallucination detection in summarization by manually annotating sentence-level faithfulness labels in LLM-generated summaries of Japanese documents. |
| Outcome: | The proposed model can detect hallucinations in Japanese documents by annotating faithfulness labels in Japanese summaries. |
From Problem-Solving to Teaching Problem-Solving: Aligning LLMs with Pedagogy using Reinforcement Learning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) are often optimized for direct question-answering, but their effectiveness is often undermined by strategically withholding answers. |
| Approach: | They propose an online reinforcement learning-based alignment framework that can quickly adapt LLMs into effective tutors using simulated student-tutor interactions. |
| Outcome: | The proposed model outperforms proprietary models like LearnLM and can be used to enhance interpretability and pedagogical quality. |
No Size Fits All: The Perils and Pitfalls of Leveraging LLMs Vary with Company Size (2025.coling-industry)
Copied to clipboard
Ashok Urlana, Charaka Vinayak Kumar, Bala Mallikarjunarao Garlapati, Ajeet Kumar Singh, Rahul Mishra
| Challenge: | Large language models are playing a pivotal role in deploying strategic use cases across organizations . lack of studies examining potential challenges and risks associated with LLMs . |
| Approach: | They propose a case study and a practical guide for industries to utilize LLMs more efficiently. |
| Outcome: | The proposed study examines the literature on large language models with industry practitioners and examines industrial publications to address these questions. |
IssueBench: Millions of Realistic Prompts for Measuring Issue Bias in LLM Writing Assistance (2026.tacl-1)
Copied to clipboard
Paul Röttger, Musashi Hinck, Valentin Hofmann, Kobi Hackenburg, Valentina Pyatkin, Faeze Brahman, Dirk Hovy
| Challenge: | Large language models are helping millions of users write texts about diverse issues . issue bias is where an LLM tends to present just one perspective on a given issue . |
| Approach: | They construct a set of 2.49m realistic English-language prompts to measure issue bias in LLM writing assistance using 3.9k templates and 212 political issues from real user interactions. |
| Outcome: | The proposed model aligns more with US Democrat than Republican voter opinion on a subset of issues. |
LLMs Underperform Graph-Based Parsers on Supervised Relation Extraction for Complex Graphs (2026.acl-short)
Copied to clipboard
| Challenge: | Relation extraction is a core NLP task which involves extracting [head, relation, dependent] RDF triples from text. |
| Approach: | They evaluate four large language models against a graph-based parser on six relation extraction datasets with sentence graphs of varying sizes and complexities. |
| Outcome: | The graph-based parser outperforms the LLMs on six relation extraction datasets with sentence graphs of varying sizes and complexities. |
Sense and Sensitivity: Examining the Influence of Semantic Recall on Long Context Code Understanding (2026.acl-long)
Copied to clipboard
| Challenge: | Large language models are increasingly used for understanding large codebases, but whether they understand operational semantics of long code context is unclear. |
| Approach: | They propose a task that achieves high semantic recall sensitivity through unpredictable operations. |
| Outcome: | The proposed task SemTrace achieves high semantic recall sensitivity through unpredictable operations. |
ScaleLLM: A Resource-Frugal LLM Serving Framework by Optimizing End-to-End Efficiency (2024.emnlp-industry)
Copied to clipboard
Yuhang Yao, Han Jin, Alay Shah, Shanshan Han, Zijian Hu, Dimitris Stripelis, Yide Ran, Zhaozhuo Xu, Salman Avestimehr, Chaoyang He
| Challenge: | Large language models (LLMs) are widely used in commercial applications . low latency is crucial due to system latency, query concurrency, and computational resources constraints. |
| Approach: | They propose a system that can be resource-efficiently served by addressing bottlenecks beyond LLM inference . they propose 4.3 speed up over vLLM and 1.5 higher throughput . |
| Outcome: | The proposed system outperforms state-of-the-arts with 1.5 higher throughput . it achieves 4.3 speed up with 64 concurrent requests on Mixtral 8x7B . |
Analogical Structure, Minimal Contextual Cues and Contrastive Distractors: Input Design for Sample-Efficient Linguistic Rule Induction (2026.eacl-long)
Copied to clipboard
| Challenge: | Recent systems that use analogical reasoning require extensive knowledge engineering and even transformer-based models show inconsistent results across complexity levels. |
| Approach: | They propose to implement analogical structure, contrastive learning, and minimal contextual cue principles into large language models that train on English verb alternations. |
| Outcome: | The proposed models learn the alternation rules with high F1 on English verb alternations. |
Walia-LLM: Enhancing Amharic-LLaMA by Integrating Task-Specific and Generative Datasets (2024.findings-emnlp)
Copied to clipboard
Israel Azime, Atnafu Tonja, Tadesse Belay, Mitiku Yohannes Fuge, Aman Wassie, Eyasu Jada, Yonas Chanie, Walelign Sewunetie, Seid Yimam
| Challenge: | Low-resource languages are left behind due to the unavailability of resources. |
| Approach: | They propose to integrate task-specific and generative datasets to improve language model performance for Amharic by fine-tuning an Amharican instruction fine-to-tuned model. |
| Outcome: | The proposed model shows promising results in different NLP tasks and compares translated instruction datasets with the original model. |
Learning to repair: Repairing model output errors after deployment using a dynamic memory of feedback (2022.findings-naacl)
Copied to clipboard
| Challenge: | Our approach pairs an LM with a growing memory of cases where the user identified an output error and provided general feedback on how to correct it. |
| Approach: | They propose to use an existing script generator to train a model to repair output errors without retraining. |
| Outcome: | The proposed model learns to apply user feedback to repair output errors while avoiding similar past mistakes on new, unseen examples. |
X-Instruction: Aligning Language Model in Low-resource Languages with Self-curated Cross-lingual Instructions (2024.findings-acl)
Copied to clipboard
| Challenge: | Large language models respond well in high-resource languages but struggle in low-resourced languages. |
| Approach: | They propose a method to construct cross-lingual instruction following samples with instruction in English and response in low-resource languages. |
| Outcome: | The proposed method builds a large-scale cross-lingual instruction tuning dataset on 10 languages. |
CRPO: Confidence-Reward Driven Preference Optimization for Machine Translation (2025.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) have shown great potential in natural language processing tasks, but their application to machine translation remains challenging due to pretraining on predominantly English-centric datasets. |
| Approach: | They propose a method that combines reward scores with model confidence to improve model selection for fine-tuning. |
| Outcome: | The proposed method outperforms existing methods in translation accuracy and data efficiency. |
On Fake News Detection with LLM Enhanced Semantics Mining (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for detecting fake news use only news embeddings to capture the lexical semantics between tokens. |
| Approach: | They propose a topic-based model with prompts to extract news embeddings from LLMs and a generalized page-rank model to extract local and global semantics. |
| Outcome: | The proposed model shows superior performance on five benchmark datasets over seven baseline methods. |
MedAgents: Large Language Models as Collaborators for Zero-shot Medical Reasoning (2024.findings-acl)
Copied to clipboard
Xiangru Tang, Anni Zou, Zhuosheng Zhang, Ziming Li, Yilun Zhao, Xingyao Zhang, Arman Cohan, Mark Gerstein
| Challenge: | Large language models face unique challenges such as domain-specific terminologies and reasoning over specialized knowledge. |
| Approach: | They propose a multi-disciplinary collaboration framework that leverages LLM-based agents in a role-playing setting. |
| Outcome: | The proposed framework excels at mining and harnessing medical expertise within LLMs, as well as extending its reasoning abilities. |
A Static Evaluation of Code Completion by Large Language Models (2023.acl-industry)
Copied to clipboard
Hantian Ding, Varun Kumar, Yuchen Tian, Zijian Wang, Rob Kwiatkowski, Xiaopeng Li, Murali Krishna Ramanathan, Baishakhi Ray, Parminder Bhatia, Sudipta Sengupta
| Challenge: | Large language models trained on code have shown great potential to increase productivity of software developers. |
| Approach: | They propose a static evaluation framework to quantify static errors in Python code completions by leveraging Abstract Syntax Trees. |
| Outcome: | The proposed framework is more efficient and applicable to code in the wild. |
Selective Span-Level Unlearning for Large Language Models (2026.acl-short)
Copied to clipboard
| Challenge: | Existing selective methods that focus on identifying token-level or span-level unlearning targets are misaligning unlearning objectives with the model’s internal behavior. |
| Approach: | They propose a selective method that uses model-intrinsic information to identify token-level or span-level unlearning targets within a text rather than entire sequences. |
| Outcome: | The proposed method achieves comparable unlearning performance while significantly better preserving retained knowledge. |
Probing the Geometry of Truth: Consistency and Generalization of Truth Directions in LLMs Across Logical Transformations and Question Answering Tasks (2025.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) are trained on vast corpora that contain substantial knowledge but their outputs often contain confidently stated inaccuracies. |
| Approach: | They propose to encode truthfulness as a distinct linear feature, termed the "truth direction", which can classify truthfulness reliably. |
| Outcome: | The proposed model can generalize to logical transformations, question-answering tasks, in-context learning, and external knowledge sources. |
Progressive Fine-Tuning for Cost-Effective Structured Attribute Generation in E-commerce (2026.acl-industry)
Copied to clipboard
| Challenge: | Large language models excel at structured information generation but face cost and latency challenges when deployed at scale in user-facing products. |
| Approach: | They propose a parameter efficient supervised fine-tuning pipeline for adapting a small language model to structured attribute generation in e-commerce product listing. |
| Outcome: | The proposed model reduces inference costs by 98% and latency by 70% on a large-scale product listing service while preserving an 86.4% user acceptance rate. |
Solving Data-centric Tasks using Large Language Models (2024.findings-naacl)
Copied to clipboard
Shraddha Barke, Christian Poelitz, Carina Negreanu, Benjamin Zorn, José Cambronero, Andrew Gordon, Vu Le, Elnaz Nouri, Nadia Polikarpova, Advait Sarkar, Brian Slininger, Neil Toronto, Jack Williams
| Challenge: | Large language models are increasingly useful for data-centric tasks, but how do we decide how much data to include in the prompt? |
| Approach: | They propose a cluster-then-select prompting technique that adds the most representative rows from the input data to the LLM prompt. |
| Outcome: | The proposed technique outperforms a baseline for tasks with syntactic variation in the input table. |
Can Post-Training Quantization Benefit from an Additional QLoRA Integration? (2025.naacl-industry)
Copied to clipboard
| Challenge: | Large language models require considerable computing resources, which can be costly and often unavailable. |
| Approach: | They propose to integrate 4-bit Post-training Quantization with QLoRA to address these issues . they demonstrate that the integration outperforms standard quantization and fine-tuning . |
| Outcome: | The proposed integration outperforms standard PTQ and 16-bit full-parameter fine-tuning on LLMs. |
On Measuring Social Biases in Prompt-Based Multi-Task Learning (2022.findings-naacl)
Copied to clipboard
| Challenge: | a large body of work within prompt engineering attempts to understand the effects of input forms and prompts in achieving superior performance. |
| Approach: | They propose a large-scale text-to-text language model trained using prompts . they consider two different forms of semantically equivalent inputs - question-answer format and premise-hypothesis format . |
| Outcome: | The proposed model can generalize into novel forms of language and handle novel tasks. |
Sample Design Engineering: An Empirical Study on Designing Better Fine-Tuning Samples for Information Extraction with LLMs (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Prompt Engineering (PE) is renowned for improving IE performance through prompt modifications, but the realm of sample design for downstream fine-tuning remains unexplored. |
| Approach: | They propose a methodical approach to enhancing LLMs’ post-tuning performance by refining input, output, and reasoning designs. |
| Outcome: | The proposed approach outperforms heuristic design strategies on three complex IE tasks with four additional LLMs. |
GeoGPT4V: Towards Geometric Multi-modal Large Language Models with Geometric Image Generation (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing datasets are too challenging for direct model learning or suffer from misalignment between text and images. |
| Approach: | They propose a pipeline that leverages GPT-4 and GPT4V to generate geometry problems with aligned text and images, facilitating model learning. |
| Outcome: | The proposed pipeline generates 4.9K geometry problems with aligned text and images, facilitating model learning. |
Mitigating Geospatial Knowledge Hallucination in Large Language Models: Benchmarking and Dynamic Factuality Aligning (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) have extensive world knowledge, but often generate inaccurate geospatial knowledge. |
| Approach: | They propose a framework for evaluation of large language models to mitigate hallucinations . they use Kahneman-Tversky Optimization to align LLMs with their reality . |
| Outcome: | The proposed evaluation framework uncovers hallucinations in 20 advanced LLMs. |
Self-Edit: Fault-Aware Code Editor for Code Generation (2023.acl-long)
Copied to clipboard
| Challenge: | Existing Large language models (LLMs) have low pass rates and accuracy on competitive programming tasks. |
| Approach: | They propose a generate-and-edit approach that uses execution results of generated code from LLMs to improve code quality on competitive programming tasks. |
| Outcome: | The proposed method improves pass@1 by 89% on APPS-dev, 31% on apps-test, and 48% on HumanEval over nine popular code generation LLMs with parameter sizes ranging from 110M to 175B. |
CodeGuard: Improving LLM Guardrails in CS Education (2026.findings-eacl)
Copied to clipboard
| Challenge: | Large language models (LLMs) are increasingly embedded in Computer Science classrooms to automate code generation, feedback, and assessment. |
| Approach: | They propose a guardrail framework for educational AI systems that can handle unsafe and irrelevant prompts. |
| Outcome: | The proposed framework reduces potentially harmful or policy-violating code completions by 30-65% without degrading performance on legitimate educational tasks. |
TARo: Token-level Adaptive Routing for LLM Test-time Alignment (2026.findings-acl)
Copied to clipboard
Arushi Rai, Qiang Zhang, Hanqing Zeng, Yunkai Zhang, Dipesh Tamboli, Xiangjun Fan, Zhuokai Zhao, Lizhu Zhang
| Challenge: | Large language models (LLMs) exhibit strong reasoning capabilities but typically require expensive post-training to reach high performance. |
| Approach: | They propose to use token-level Adaptive Routing to steer frozen LLMs toward structured reasoning entirely at inference time. |
| Outcome: | Extensive experiments show that TARo significantly improves reasoning performance by up to +22.4% over base model and +8.4% . |
OVM, Outcome-supervised Value Models for Planning in Mathematical Reasoning (2024.findings-naacl)
Copied to clipboard
| Challenge: | Large language models (LLMs) struggle with maintaining accuracy throughout multiple reasoning steps, especially in mathematical reasoning where an error in earlier steps can propagate to subsequent ones and ultimately leading to an incorrect answer. |
| Approach: | They propose an Outcome-supervised Value Model (OVM) that employs outcome supervision for training a value model, which prioritizes steps that lead to accurate conclusions. |
| Outcome: | The proposed model performs better on two multi-step reasoning datasets, GSM8K and Game of 24. |
SPAGBias: Uncovering and Tracing Structured Spatial Gender Bias in Large Language Models (2026.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) are being used in urban planning but there is concern that they reproduce or amplify such biases. |
| Approach: | They propose a framework to evaluate spatial gender bias in large language models . they use a taxonomy of 62 urban micro-spaces, a prompt library and three diagnostic layers . |
| Outcome: | The proposed framework identifies structured gender-space associations that go beyond the public-private divide, forming nuanced micro-level mappings. |
CarExpert: Leveraging Large Language Models for In-Car Conversational Question Answering (2023.emnlp-industry)
Copied to clipboard
Md Rashad Al Hasan Rony, Christian Suess, Sinchana Ramakanth Bhat, Viju Sudhi, Julia Schneider, Maximilian Vogel, Roman Teucher, Ken Friedl, Soumya Sahoo
| Challenge: | Large language models (LLMs) have demonstrated remarkable performance by following natural language instructions without fine-tuning them on domain-specific tasks and data. |
| Approach: | They propose an in-car retrieval-augmented conversational question-answering system that uses large language models to generate natural, safe and domain-specific answers. |
| Outcome: | The proposed system outperforms state-of-the-art LLMs in generating safe and domain-specific answers. |
DisComp: A Two-Stage Prompt Optimization Framework Combining Task-Agnostic and Task-Aware Compression (2025.findings-naacl)
Copied to clipboard
| Challenge: | Extended prompts can lead to substantial computational overhead and increased hardware demands, limiting the scalability and efficiency of large language models. |
| Approach: | They propose a two-stage prompt compression framework that combines task-agnostic and task-based strategies to efficiently compress prompt length without compromising performance. |
| Outcome: | The proposed framework outperforms task-agnostic and task-specific compression methods on three benchmark datasets and is up to 6.56 faster at inference compared to the best token-level compression method. |
English as Defense Proxy: Mitigating Multilingual Jailbreak via Eliciting English Safety Knowledge (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models excel in many tasks, but their safety guarantees vary by language. |
| Approach: | They propose a unified approach that leverages English as a universal safety anchor. |
| Outcome: | The proposed approach leverages English as defense proxy (E-Proxy) to transfer safety knowledge across languages. |
A Shared Geometry of Difficulty in Multilingual Language Models (2026.acl-short)
Copied to clipboard
| Challenge: | Large language models encode problem difficulty as an internal signal that can be linearly decoded from their residuals. |
| Approach: | They train linear probes on the AMC subset of the Easy2Hard benchmark, translated into 21 languages, and found difficulty-related signals emerge at two distinct stages of the model internals. |
| Outcome: | The results show that difficulty-related signals emerge at two distinct stages of the model internals, corresponding to shallow (early-layers) and deep (later-layer) representations, that exhibit functionally different behaviors. |
Provably Confidential Language Modelling (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing methods to train language models without memorizing sensitive data are mismatched and can be difficult to screen and filter. |
| Approach: | They propose a method to train language generation models while protecting the confidential segments of training data. |
| Outcome: | The proposed method prevents unintended memorization by randomizing parts of the training process while protecting strong confidentiality. |
Can Large Language Models Generate High-quality Patent Claims? (2025.findings-naacl)
Copied to clipboard
| Challenge: | Large language models (LLMs) have shown exceptional performance across various text generation tasks, but remain under-explored in the patent domain, which offers highly structured and precise language. |
| Approach: | They construct a dataset to investigate the performance of current LLMs in patent claim generation. |
| Outcome: | The proposed model outperforms state-of-the-art general LLMs in patent claim generation. |
ExAnte: A Benchmark for Ex-Ante Inference in Large Language Models (2026.eacl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) struggle with ex-ante reasoning—making inferences or predictions without access to future information. |
| Approach: | They propose a benchmark that assesses LLMs’ ex-ante inference ability across four tasks: stock prediction, question answering, Wikipedia event generation, and scientific publication generation. |
| Outcome: | The proposed benchmark assesses LLMs’ ex-ante inference ability across four tasks. |
Self-Debiasing Large Language Models: Zero-Shot Recognition and Reduction of Stereotypes (2025.naacl-short)
Copied to clipboard
Isabel O. Gallegos, Ryan Aponte, Ryan A. Rossi, Joe Barrow, Mehrab Tanjim, Tong Yu, Hanieh Deilamsalehy, Ruiyi Zhang, Sungchul Kim, Franck Dernoncourt, Nedim Lipka, Deonna Owens, Jiuxiang Gu
| Challenge: | Large language models exhibit harmful social biases, but they are often difficult to train and modify. |
| Approach: | They leverage the zero-shot capabilities of large language models to reduce stereotyping . they introduce a technique called zero- shot self-debiasing to reduce bias . |
| Outcome: | The proposed technique reduces stereotyping across nine different social groups while relying on the LLM itself and a simple prompt. |
EvoEdit: Evolving Null-space Alignment for Robust and Efficient Knowledge Editing (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to modifying large language models require continual updates to rectify outdated or erroneous knowledge. |
| Approach: | They propose a model editing strategy that mitigates catastrophic interference through sequential null-space alignment. |
| Outcome: | EvoEdit achieves better or comparable performance than prior state-of-the-art techniques with up to 3.53 speedup. |
RoZO: Geometry-Aware Zeroth-Order Fine-Tuning on Low-Rank Adapters for Black-Box Large Language Models (2026.eacl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated exceptional performance across a wide range of tasks, yet fine-tuning them efficiently under black-box or memory-constrained settings remains challenging. |
| Approach: | They propose a Riemannian zeroth-order optimization framework that constrains updates to the tangent space of the LoRA manifold. |
| Outcome: | The proposed framework achieves more stable convergence, tighter variance bounds, and superior performance compared to existing ZO methods. |
M4: Multi-generator, Multi-domain, and Multi-lingual Black-Box Machine-Generated Text Detection (2024.eacl-long)
Copied to clipboard
Yuxia Wang, Jonibek Mansurov, Petar Ivanov, Jinyan Su, Artem Shelmanov, Akim Tsvigun, Chenxi Whitehouse, Osama Mohammed Afzal, Tarek Mahmoud, Toru Sasaki, Thomas Arnold, Alham Aji, Nizar Habash, Iryna Gurevych, Preslav Nakov
| Challenge: | Large language models generate fluent responses to user queries, but they are also susceptible to misuse in journalism, education, and academia. |
| Approach: | They propose a large-scale benchmark for machine-generated text detection that is a multi-generator, multi-domain, and multi-lingual corpus. |
| Outcome: | The proposed system can detect machine-generated text and pinpoint misuse . the proposed system is based on a large-scale benchmark dataset . |
TreeCut: A Synthetic Unanswerable Math Word Problem Dataset for LLM Hallucination Evaluation (2025.acl-short)
Copied to clipboard
| Challenge: | Large language models (LLMs) can achieve near-human performance on benchmarks like GSM8K, yet their true reasoning ability remains disputed. |
| Approach: | They propose a synthetic dataset that generates infinite unanswerable math word problems and their answerable counterparts by representing each question as a tree and removing selected necessary conditions. |
| Outcome: | Experiments show TreeCut induces hallucinations in large language models, including GPT-4o and o3-mini, with rates of 64% and 44% in worst-case scenarios. |
MOOSE-Copilot: A Web-Based Interactive Assistant for Unified Exploratory and Fine-Grained Scientific Hypothesis Discovery (2026.acl-demo)
Copied to clipboard
| Challenge: | Existing approaches to scientific hypothesis discovery operate autonomously with little to no human guidance. |
| Approach: | They propose a framework that empowers scientists to steer the generative process via explicit signals. |
| Outcome: | The proposed framework outperforms existing models in terms of quality and accuracy of the input signals. |
Do LLMs Catch Their Own Mistakes? A Comprehensive Benchmark for Reflective Tool Use LLMs (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing benchmarks primarily evaluate planning and execution success, overlooking the self-reflective dimension of tool use. |
| Approach: | They propose a benchmark to assess LLMs’ self-reflective reasoning in tool-augmented multi-turn dialogues. |
| Outcome: | The proposed benchmark covers 10 domains with 88 distinct APIs and 968 annotated dialogues, systematically injecting diverse error types arising from both user and assistant behavior. |
Lil: Less is Less When Applying Post-Training Sparse-Attention Algorithms in Long-Decode Stage (2026.findings-acl)
Copied to clipboard
Junhao Hu, Fangze Li, Mingtao Xu, Feifan Meng, Shiju Zhao, Tiancheng Hu, Ting Peng, Anmin Liu, Wenrui Huang, Chenxu Liu, Ziyue Hua, Tao Xie
| Challenge: | Prior work typically decomposes inference into prefill and decode stages, with the decode stage dominating total latency. |
| Approach: | They propose an algorithm that detects threshold where information loss exceeds information gain during sparse decoding to reduce token consumption by up to 90% and a marginal accuracy degradation of less than 2%. |
| Outcome: | The proposed algorithm reduces token consumption by 90% with a marginal accuracy degradation of less than 2% across reasoning-intensive benchmarks. |
Static Models, Dynamic World: A Unified Perspective on Temporal Perception in Large Language Models (2026.findings-acl)
Copied to clipboard
Chenhao Li, Dandan Song, Changzhi Zhou, Jun Yang, Yuhang Tian, Huipeng Ma, Guangyuan Feng, Luan Zhang, Xudong Li, Ke Duan
| Challenge: | Large language models are trained on static corpora but deployed in a dynamic world . a foundational tension remains between time and the ability to understand it . |
| Approach: | They formalize temporal queries in an information-theoretic framework based on parametric reachability of temporal premises and answers. |
| Outcome: | The proposed framework formalizes temporal queries in an information-theoretic framework based on parametric reachability of temporal premises and answers . the framework induces four temporal information regimes corresponding to internal reasoning, answer recency, premise anchoring, and genuine world indeterminacy . |
Decomposed Trust: Privacy, Adversarial Robustness, Ethics, and Fairness in Low-Rank LLMs (2026.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) have driven major advances across domains, yet their massive size hinders deployment in resource-constrained settings. |
| Approach: | They propose to compress large language models to reduce computation and memory consumption while maintaining accuracy. |
| Outcome: | The proposed algorithms preserve training data privacy but weaken the protection of personally identifiable information during conversations. |
Explainable Hallucination through Natural Language Inference Mapping (2025.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) often generate hallucinated content, making it crucial to identify and quantify inconsistencies in their outputs. |
| Approach: | They propose a framework that maps entailment and contradiction relations between inputs and outputs using a natural language inference model. |
| Outcome: | The proposed framework outperforms state-of-the-art methods by five percentage points while providing clear, interpretable explanations. |
Through the Looking Glass of Multilingual AI: Contrasting Language- and Name Script-Dependent Ethnic Hierarchies in GPT and DeepSeek (2026.acl-srw)
Copied to clipboard
| Challenge: | a recent study found that large language models are biased overwhelmingly Anglocentric . a stereotype perceptual map is a framework for analyzing how ethnic groups are positioned along evaluative dimensions. |
| Approach: | They use a stereotype perceptual map to examine how ethnic groups are positioned along evaluative dimensions. |
| Outcome: | The stereotype perceptual map analyzes model behavior across languages, scripts, evaluative domains and models. |
Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus (2021.emnlp-main)
Copied to clipboard
Jesse Dodge, Maarten Sap, Ana Marasović, William Agnew, Gabriel Ilharco, Dirk Groeneveld, Margaret Mitchell, Matt Gardner
| Challenge: | Large text corpora are often introduced with minimal documentation . documenting collection process, composition, intended uses, and other are key for structured, task-specific datasets. |
| Approach: | They propose to document a dataset created by applying filters to a single snapshot of Common Crawl. |
| Outcome: | The proposed dataset shows that blocklist filtering removes text from minority individuals and patents. |
MenatQA: A New Dataset for Testing the Temporal Comprehension and Reasoning Abilities of Large Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) have shown nearly saturated performance on many NLP tasks. |
| Approach: | They construct multiple sensitive factors time QA which encompasses three temporal factors . they test current mainstream LLMs with different parameter sizes . |
| Outcome: | The proposed model incorporates three temporal factors with 2,853 samples . the results show that LLMs fall behind smaller models on these factors . |
Measuring Bias and Agreement in Large Language Model Presupposition Judgments (2025.findings-acl)
Copied to clipboard
| Challenge: | Identifying linguistic bias in text requires the identification of explicit statements and presuppositions . large language models can be used to detect subtle forms of bias with no clear lexical signals . |
| Approach: | They propose to prompt large language models to evaluate presuppositions across texts . they find that LLMs may inadvertently reflect societal biases when identifying presuposed content . |
| Outcome: | The proposed model can be used to detect linguistic biases in text, but its accuracy is unclear . linguistic factors associated with human-model alignment suggest biase influenced by gender and ideology. |
FuxiTranyu: A Multilingual Large Language Model Trained with Balanced Data (2024.emnlp-industry)
Copied to clipboard
Haoran Sun, Renren Jin, Shaoyang Xu, Leiyu Pan, null Supryadi, Menglong Cui, Jiangcun Du, Yikun Lei, Lei Yang, Ling Shi, Juesi Xiao, Shaolin Zhu, Deyi Xiong
| Challenge: | Large language models exhibit significant performance discrepancies between high- and low-resource languages. |
| Approach: | They present an open-source multilingual LLM with 8 billion parameters and a multilingual instruction dataset. |
| Outcome: | The proposed model achieves consistent multilingual representations across languages. |
Adaptive and Representative Multi-Interest Modeling for Recommendation with Large Language Model (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for multi-interest analysis of users rely on heuristic assumptions . however, the granularity of raw generation of LLMs is agnostic, leading to overly fine or coarse interest grouping. |
| Approach: | They propose an LLM-driven adaptive and representative multi-interest modeling framework that exploits the agnostic granularity of LLMs for multi-interest analysis. |
| Outcome: | The proposed model outperforms baselines on real-world datasets. |
Summarizing, Simplifying, and Synthesizing Medical Evidence using GPT-3 (with Varying Success) (2023.acl-short)
Copied to clipboard
| Challenge: | Large language models are capable of producing high quality summaries of general domain news articles in few- and zero-shot settings, but it is unclear whether they are similarly capable in more specialized domains such as biomedicine. |
| Approach: | They use GPT-3 to generate single- and multi-document summaries of biomedical articles, given no supervision, using a set of annotations. |
| Outcome: | The proposed model outperforms fully supervised models in generic news summarization, but struggles to synthesize evidence across multiple documents. |
SocialBench: Sociality Evaluation of Role-Playing Conversational Agents (2024.findings-acl)
Copied to clipboard
Hongzhan Chen, Hehong Chen, Ming Yan, Wenshen Xu, Gao Xing, Weizhou Shen, Xiaojun Quan, Chenliang Li, Ji Zhang, Fei Huang
| Challenge: | Existing studies on role-playing agents have focused on enhancing their conversational capability, role-specific knowledge and style, but there has been a gap in assessing their social intelligence. |
| Approach: | They propose a benchmark to evaluate the sociality of role-playing agents using LLMs. |
| Outcome: | The proposed benchmark is constructed from various sources and covers a wide range of 500 characters and over 6,000 question prompts and 30,800 multi-turn role-playing utterances. |
LTRAG: Enhancing Autoformalization and Self-refinement for Logical Reasoning with Thought-Guided RAG (2025.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) have shown promise in natural language reasoning, especially with techniques like chain-of-thought prompting. |
| Approach: | They propose a framework to enhance autoformalization and self-refinement for logical reasoning with Retrieval-Augmented Generation (RAG) by building knowledge bases of thought-guided examples. |
| Outcome: | The proposed framework outperforms Logic-LM and LINC on FOLIO and AR-LSAT, and achieves an accuracy gain of 13% over Logic LM and the proposed methods on GPT-4 and AR LSAT. |
CRT-QA: A Dataset of Complex Reasoning Question Answering over Tabular Data (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) show powerful reasoning abilities on text-based tasks, but their reasoning capability on structured data such as tables has not been systematically explored. |
| Approach: | They first establish a comprehensive taxonomy of reasoning and operation types for tabular data analysis and then construct a complex reasoning QA dataset over tabular dataset. |
| Outcome: | The proposed method is able to solve table reasoning tasks without handcrafted demonstrations. |
Beyond Divergent Creativity: A Human-Based Evaluation of Creativity in Large Language Models (2026.findings-eacl)
Copied to clipboard
| Challenge: | Large language models are increasingly used in verbal creative tasks. |
| Approach: | They propose a divergent association task that focuses on novelty, ignoring appropriateness, a core component of creativity. |
| Outcome: | The proposed model scores are lower than baselines with no creative abilities, undermining its validity for model evaluation. |
Zero-knowledge LLM hallucination detection and mitigation through fine-grained cross-model consistency (2025.emnlp-industry)
Copied to clipboard
| Challenge: | Existing methods for hallucination management fail to integrate both detection and mitigation without external knowledge sources. |
| Approach: | They propose a black-box framework that leverages fine-grained cross-model consistency to detect and mitigate hallucinations in LLM outputs without external knowledge sources. |
| Outcome: | The proposed framework improves hallucination detection scores by 6-39% on a FELM dataset . it achieves 9 percentage points improvement in answer accuracy on the GPQA-diamond dataset compared to existing approaches . |
How Alignment and Jailbreak Work: Explain LLM Safety through Intermediate Hidden States (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) rely on safety alignment to avoid malicious user inputs. |
| Approach: | They employ weak classifiers to explain LLM safety through the intermediate hidden states. |
| Outcome: | The proposed model can identify malicious and normal inputs and detect malicious ones without jailbreak. |
Attack Prompt Generation for Red Teaming and Defending Large Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies construct attack prompts via manual or automatic methods, but these methods have limitations on cost and quality. |
| Approach: | They propose an attack framework to instruct LLMs to mimic human-generated prompts through in-context learning and a defense framework that fine-tunes victim LLM's through iterative interactions with the attack framework. |
| Outcome: | The proposed approach is based on experiments on different LLMs to evaluate their effectiveness against red teaming attacks. |
MathGenie: Generating Synthetic Data with Question Back-translation for Enhancing Mathematical Reasoning of LLMs (2024.acl-long)
Copied to clipboard
| Challenge: | Existing models have demonstrated outstanding capabilities in mathematical reasoning, but there is a performance gap between open-source models and closed-source ones. |
| Approach: | They propose a method for generating diverse and reliable math problems by leveraging the ground-truth solutions of the seed data. |
| Outcome: | The proposed model outperforms open-source models across five representative mathematical reasoning datasets. |
On Context Utilization in Summarization with Large Language Models (2024.acl-long)
Copied to clipboard
| Challenge: | Large language models excel in abstractive summarization tasks, delivering fluent and pertinent summaries. |
| Approach: | They conduct the first comprehensive study on context utilization and position bias in summarization. |
| Outcome: | The proposed benchmark compares two methods to alleviate position bias in summarization tasks. |
LLMs Are Not Intelligent Thinkers: Introducing Mathematical Topic Tree Benchmark for Comprehensive Evaluation of LLMs (2025.naacl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) have impressive capabilities in mathematical reasoning, but their effectiveness is limited to specific mathematical topics. |
| Approach: | They propose to use the MaTT benchmark to assess large language models' accuracy in multiple-choice scenarios. |
| Outcome: | The proposed model achieved 54% accuracy in a multiple-choice scenario, while the Chain-of-Thought prompting did not improve. |
How Does Response Length Affect Long-Form Factuality (2025.findings-acl)
Copied to clipboard
| Challenge: | Despite growing attention to LLM factuality, the effect of response length on factual accuracy remains underexplored. |
| Approach: | They propose an automatic and bi-level long-form factuality evaluation framework which achieves high agreement with human annotations while being cost-effective. |
| Outcome: | The proposed framework achieves high agreement with human annotations while being cost-effective. |
Alignment-Enhanced Decoding: Defending Jailbreaks via Token-Level Adaptive Refining of Probability Distributions (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing defenses against jailbreaks focus on perturbing or inspecting inputs, but ignore competing objectives, the underlying cause of alignment failures. |
| Approach: | They propose a novel defense that employs adaptive decoding to address the root causes of jailbreak issues. |
| Outcome: | The proposed defense improves safety alignment while maintaining helpfulness. |
Direct Evaluation of Chain-of-Thought in Multi-hop Reasoning with Knowledge Graphs (2024.findings-acl)
Copied to clipboard
| Challenge: | Prior research on evaluating large language models focused on answer accuracy, neglecting the correctness of the generated CoT. |
| Approach: | They propose a discriminative and generative CoT evaluation paradigm to assess LLMs’ knowledge of reasoning and the accuracy of the generated CoT. |
| Outcome: | The proposed evaluation paradigm assesses LLMs’ knowledge of reasoning and the accuracy of the generated CoT. |
Unfamiliar Finetuning Examples Control How Language Models Hallucinate (2025.naacl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) generate plausible-sounding responses that are factually incorrect. |
| Approach: | They propose an approach to learn more reliable reward models by modifying how unfamiliar finetuning examples are supervised to influence model responses to unfamiliar queries. |
| Outcome: | The proposed approach improves the efficacy of RL factuality finetuning in long-form biography and book/movie plot generation tasks. |
Topology-of-Question-Decomposition: Enhancing Large Language Models with Information Retrieval for Knowledge-Intensive Tasks (2025.coling-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) are constrained to chaining immediate reasoning steps and relying solely on parametric knowledge. |
| Approach: | They propose a framework that activates retrieval only when necessary to improve answer accuracy. |
| Outcome: | Experiments show that the proposed framework improves performance in knowledge-intensive tasks. |
Deliberative Searcher: Improving LLM Reliability via Reinforcement Learning with Constraints (2026.acl-long)
Copied to clipboard
| Challenge: | Large language models with search capabilities often exhibit miscalibrated confidence, causing incorrect answers with high certainty. |
| Approach: | They propose a reasoning-primary framework that integrates search operations into chain-of-thought generation while maintaining explicit confidence calibration. |
| Outcome: | The proposed framework improves accuracy and reliability of large language models with search capabilities. |
Task Matters: Knowledge Requirements Shape LLM Responses to Context–Memory Conflict (2026.findings-acl)
Copied to clipboard
| Challenge: | Prior work has shown that large language models favor parametric knowledge under conflict, but this setting assumes that tasks should always rely on the provided passage. |
| Approach: | They propose a model-agnostic diagnostic framework that holds underlying knowledge constant while injecting controlled conflicts across tasks with varying knowledge requirements. |
| Outcome: | Evaluating representative open-source LLMs, the proposed framework holds underlying knowledge constant while injecting controlled conflicts across tasks with varying knowledge requirements. |
Enhancing Agent Learning through World Dynamics Modeling (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods assume that large language models have a complete understanding of their environment, overlooking potential gaps in their grasp of actual world dynamics. |
| Approach: | They propose a framework that discovers world dynamics from a small number of demonstrations, verifies the correctness of these dynamics, and evolves new, advanced dynamics tailored to the current situation. |
| Outcome: | The proposed framework discovers, verifies, and evolves world dynamics from a small number of demonstrations, and compares the automatically generated dynamics with human-annotated world dynamics. |
Improving Zero-shot Reader by Reducing Distractions from Irrelevant Documents in Open-Domain Question Answering (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) enable zero-shot approaches in open domain question answering (ODQA), yet with limited advancements as the reader is compared to the retriever. |
| Approach: | They propose to use a distraction-aware answer selection framework to mitigate the impact of irrelevant documents in the retrieved set and the overconfidence of the generated answers to enhance the performance of zero-shot readers. |
| Outcome: | The proposed approach handles distraction across diverse scenarios, enhancing the performance of zero-shot readers. |
Knowledgeable In-Context Tuning: Exploring and Exploiting Factual Knowledge for In-Context Learning (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing studies have explored multiple aspects that affect the performance of large language models (LLMs) such as input-output mapping, extensive data resources, and the ability to train on labeled examples. |
| Approach: | They propose a framework that injects knowledge into LLMs during continual self-supervised pre-training and judiciously selects examples with high knowledge relevance. |
| Outcome: | The proposed framework outperforms baseline models and improves by more than 13% and 7% on text classification and question-answering tasks. |
A User-Centric Multi-Intent Benchmark for Evaluating Large Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing benchmarks focus on specific predefined model abilities, such as world knowledge, reasoning, etc., making it difficult for users to determine which LLM best suits their particular needs. |
| Approach: | They propose to evaluate large language models from a user-centric perspective and use real-world use cases to identify their effectiveness under distinct intents. |
| Outcome: | The proposed benchmarks achieve a correlation between human preference and the user-reported scenarios and human intents. |
Word Surprisal Correlates with Sentential Contradiction in LLMs (2026.eacl-long)
Copied to clipboard
| Challenge: | Existing models are primarily optimized for task-specific performance, lacking well-defined objectives or linguistic grounding. |
| Approach: | They propose a token-to-word decoding algorithm that extends theoretically grounded probability estimation to open-vocabulary settings. |
| Outcome: | The proposed algorithm can localize sentence-level inconsistency at the word level, establishing a quantitative link between lexical uncertainty and sentential semantics. |
Dancing Between Success and Failure: Edit-level Simplification Evaluation using SALSA (2023.emnlp-main)
Copied to clipboard
| Challenge: | Traditional human evaluation methods for text simplification often relies on individual, shallow sentence-level ratings, easily affected by the annotator's preference or bias. |
| Approach: | They propose an edit-based human annotation framework that enables holistic and fine-grained text simplification evaluation. |
| Outcome: | The proposed framework is able to predict sentence- and word-level quality simultaneously and report promising results. |
PPL-MCTS: Constrained Textual Generation Through Discriminator-Guided MCTS Decoding (2022.naacl-main)
Copied to clipboard
| Challenge: | Large language models (LM) based on transformers generate plausible long texts . a discriminator-guided approach allows to apply constraints more finely and dynamically. |
| Approach: | They propose to use a discriminator-guided approach to generate constrained texts without fine-tuning the LM. |
| Outcome: | The proposed method is easier and cheaper to train than fine-tuning the LM. |
Sanitizing Large Language Models in Bug Detection with Data-Flow (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) have potential in code reasoning tasks but the hallucination effect can compromise the reliability of bug reports. |
| Approach: | They propose a new schema of bug detection that enforces LLMs to emit data-flow paths in few-shot chain-of-thought prompting and validates them via the program-property decomposition. |
| Outcome: | The proposed approach achieves 91.03% precision and 74.00% recall upon synthetic benchmarks and boosts precision by 21.99% with the sanitization. |
Steering Large Language Models for Machine Translation Personalization (2026.eacl-long)
Copied to clipboard
| Challenge: | Recent advances in interpretability research have highlighted the effectiveness of steering methods for MT personalization. |
| Approach: | They examine steering strategies for personalizing automatic translations when few examples are available. |
| Outcome: | The proposed steering methods yield higher inference-time computational efficiency than prompting approaches. |
Perception Compressor: A Training-Free Prompt Compression Framework in Long Context Scenarios (2025.findings-naacl)
Copied to clipboard
| Challenge: | Long prompts contain redundant information and are sensitive to the position of key information in long context scenarios. |
| Approach: | They propose a training-free prompt compression framework that retains key information at token level while removing distracting tokens. |
| Outcome: | The proposed framework outperforms existing methods on long context benchmarks. |
Improving In-Context Learning with Prediction Feedback for Sentiment Analysis (2024.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) have achieved promising results in sentiment analysis through the in-context learning paradigm. |
| Approach: | They propose a framework that incorporates prior predictions and feedback to improve sentiment understanding by incorporating prior feedback and leveraging a feedback-driven prompt. |
| Outcome: | The proposed framework improves on nine sentiment analysis datasets with an average improvement of 5.95% over conventional methods. |
Persuasion at Play: Understanding Misinformation Dynamics in Demographic-Aware Human-LLM Interactions (2026.eacl-long)
Copied to clipboard
| Challenge: | Existing challenges in misinformation exposure and susceptibility vary across demographics. |
| Approach: | They propose a framework that investigates the bidirectional persuasion dynamics between LLMs and humans when exposed to misinformation. |
| Outcome: | The proposed framework analyzes the spread of misinformation under persuasion among demographic-oriented LLM agents. |
When Do Pre-Training Biases Propagate to Downstream Tasks? A Case Study in Text Summarization (2023.eacl-main)
Copied to clipboard
Faisal Ladhak, Esin Durmus, Mirac Suzgun, Tianyi Zhang, Dan Jurafsky, Kathleen McKeown, Tatsunori Hashimoto
| Challenge: | Existing studies have shown that large language models contain linguistic and societal biases, but it is unclear how these biase amplify to downstream tasks. |
| Approach: | They investigate how name-nationality bias propagates from pre-training to downstream tasks . they show that these biases manifest themselves as hallucinations in summarization . |
| Outcome: | The proposed model can reduce the rate of hallucinations, but does not change the types of biases that do appear. |
Scalable Data Synthesis through Human-like Cognitive Imitation and Data Recombination (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) rely on massive amounts of training data, however, the quantity of empirically observed data is limited. |
| Approach: | They propose a data synthesis framework that mimics human cognitive behaviors by recombining and interconnecting heterogeneous data from diverse sources. |
| Outcome: | The proposed framework mimics human cognitive behaviors by recombining and interconnecting heterogeneous data from diverse sources thereby enhancing advanced reasoning capabilities in large language models. |
Rectifying Demonstration Shortcut in In-Context Learning (2024.naacl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) can solve tasks with a few demonstrations, but often rely on their pre-trained semantic priors rather than the input-label relationships to proceed with ICL prediction. |
| Approach: | They propose a demonstration-aware calibration method to improve LLMs' ability to learn new input-label relationships from demonstrations. |
| Outcome: | The proposed method improves the original ICL task and the task learning setting, and the results are generalized across three LLM families. |
Cross-Cultural Transfer of Commonsense Reasoning in LLMs: Evidence from the Arab World (2025.findings-emnlp)
Copied to clipboard
| Challenge: | a recent study examined the potential for cross-cultural transfer of commonsense reasoning . merely 12 culture-specific examples from one country can improve performance in others by 10% on average . |
| Approach: | They evaluate cross-cultural transfer of commonsense reasoning within the arab world . they use in-context learning and demonstration-based reinforcement to evaluate alignment methods . |
| Outcome: | The proposed model can improve performance in cultures with cultural similarities in the Arab world by 10% on average. |
Mitigating Selection Bias with Node Pruning and Auxiliary Options (2025.acl-long)
Copied to clipboard
| Challenge: | Large language models exhibit systematic preferences for answer choices when answering multiple-choice questions. |
| Approach: | They propose two methods to identify and remove internal sources of selection bias . they propose Choice Kullback-Leibler Divergence (CKLD) to capture distributional imbalances in model predictions. |
| Outcome: | The proposed methods improve answer accuracy while reducing selection bias. |
SH2: Self-Highlighted Hesitation Helps You Decode More Truthfully (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) have made great progress in text generation but suffer from hallucinations during reasoning and generation. |
| Approach: | They propose an inference-time method to help LLMs decode truthfully by selecting tokens with the lowest probabilities and concatenating them to the original context. |
| Outcome: | The proposed method improves LLaMA-7b, LLama2-7b and Mistral-7b on hallucination tasks. |
Steering Knowledge Selection Behaviours in LLMs via SAE-Based Representation Engineering (2025.naacl-long)
Copied to clipboard
Yu Zhao, Alessio Devoto, Giwon Hong, Xiaotang Du, Aryo Pradipta Gema, Hongru Wang, Xuanli He, Kam-Fai Wong, Pasquale Minervini
| Challenge: | Large language models store factual knowledge in their parameters but their parametric knowledge can conflict with the information provided in the context. |
| Approach: | They propose a training-free representation engineering method that uses pre-trained sparse auto-encoders to control the knowledge selection behaviour of large language models. |
| Outcome: | The proposed method can control the use of both knowledge sources to resolve knowledge conflict in open-domain question-answering tasks surpassing existing representation engineering methods (+10%) and contrastive decoding methods (+5%). |
Dynamic Attention-Guided Context Decoding for Mitigating Context Faithfulness Hallucinations in Large Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing methods, such as a n-terminal coding, do not provide accurate data for large language models. |
| Approach: | They propose a lightweight framework that leverages attention distributions and uncertainty signals in a single-pass decoding. |
| Outcome: | Experiments on open-book QA datasets show that DAGCD improves faithfulness and robustness while preserving computational efficiency. |
Human and LLM-Based Resume Matching: An Observational Study (2025.findings-naacl)
Copied to clipboard
Swanand Vaishampayan, Hunter Leary, Yoseph Berhanu Alebachew, Louis Hickman, Brent A. Stevenor, Weston Beck, Chris Brown
| Challenge: | Resume matching assesses the extent to which candidates qualify for jobs based on the content of resumes. |
| Approach: | They compare GPT-4 and human ratings for resumes submitted to job openings from diverse fields using real-world evaluation criteria. |
| Outcome: | The proposed model improves the quality of LLM ratings and does not show bias. |
Complementary Explanations for Effective In-Context Learning (2023.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) have remarkable capabilities in learning from expla- nations in prompts, but there has been limited understanding of exactly how these explana- tions function or why they are effective. |
| Approach: | They propose a maximal marginal relevance-based exemplar selection approach to construct exemplar sets that are both relevant and comple- mentary. |
| Outcome: | The proposed model improves in- context learning performance across three tasks on multiple LLMs. |
Select-Then-Decompose: From Empirical Analysis to Adaptive Selection Strategy for Task Decomposition in Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing task decomposition methods focus on memory, tool usage, and feedback mechanisms, but they often overlook the trade-off between performance and cost. |
| Approach: | They propose a strategy that selects the most suitable decomposition approach based on task characteristics and enhances the reliability of the results through a verification module. |
| Outcome: | The proposed strategy is based on categories of approaches, characteristics of tasks, and configuration of decomposition and execution models. |
UHGEval: Benchmarking the Hallucination of Chinese Large Language Models via Unconstrained Generation (2024.acl-long)
Copied to clipboard
Xun Liang, Shichao Song, Simin Niu, Zhiyu Li, Feiyu Xiong, Bo Tang, Yezhaohui Wang, Dawei He, Cheng Peng, Zhonghao Wang, Haiying Deng
| Challenge: | Large language models (LLMs) produce hallucinated text, compromising their practical utility in professional contexts. |
| Approach: | They have developed an unconstrained hallucination generation evaluation benchmark that contains hallucines generated by large language models with minimal restrictions. |
| Outcome: | The proposed benchmarks are based on a Chinese-language dataset that is lacking in the field. |
Faithful Serum: Mitigating the Faithfulness Gap in Textual Explanations of LLM Decisions via Attribution Guidance (2026.acl-long)
Copied to clipboard
| Challenge: | Prior work has focused on generating convincing rationales that appear to be subjectively faithful, but it remains unclear whether these explanations are epistemic faithful. |
| Approach: | They propose a method that enhances epistemic faithfulness by guiding explanation generation through attention-level interventions, informed by token-level heatmaps. |
| Outcome: | The proposed method significantly improves epistemic faithfulness across multiple models, benchmarks, and prompts. |
ResearchArena: Benchmarking Large Language Models’ Ability to Collect and Organize Information as Research Agents (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models excel across many natural language processing tasks but face challenges in domain-specific, analytical tasks such as conducting research surveys. |
| Approach: | They propose a benchmark to evaluate LLMs' capabilities in conducting research surveys. |
| Outcome: | The proposed benchmark is designed to evaluate LLMs' capabilities in conducting research surveys. |
Constraining Sequential Model Editing with Editing Anchor Compression (2025.findings-naacl)
Copied to clipboard
| Challenge: | Large language models (LLMs) exhibit hallucinations due to incorrect or outdated knowledge embedded in their parameters. |
| Approach: | They propose a framework to constrain the deviation of the parameter matrix during sequential editing by selecting editing anchors that are important in encoding new relations without deviating too much from the original matrix. |
| Outcome: | The proposed framework minimizes deviations caused by model editing while retaining over 70% of the general abilities. |
CipherBank: Exploring the Boundary of LLM Reasoning Capabilities through Cryptography Challenge (2025.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated remarkable capabilities, but their capabilities in cryptographic decryption tasks remain underexplored. |
| Approach: | They propose a benchmark to evaluate the reasoning capabilities of large language models in cryptographic decryption tasks. |
| Outcome: | The proposed benchmark examines the reasoning capabilities of large language models in cryptographic decryption tasks. |
LLM Jailbreak Detection for (Almost) Free! (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for detecting jailbreak prompts entail significant computational costs . |
| Approach: | They propose a free jailbreak detection method which scales logits by temperature to detect jailbreak prompts . |
| Outcome: | The proposed method detects jailbreak prompts with no additional computational costs. |
Beyond Blind Following: Evaluating Robustness of LLM Agents under Imperfect Guidance (2026.eacl-long)
Copied to clipboard
Yao Fu, Ran Qiu, Xinhe Wang, Jacob Sansom, Sathvika Ayyappa Prabhu, Huijie Tang, Jaekyeom Kim, Sungryull Sohn, Honglak Lee
| Challenge: | Large language models (LLMs) have shown strong capabilities as task-solving agents across interactive domains, but in complex environments, auxiliary guidance may be imperfect. |
| Approach: | They propose a benchmark to measure the robustness of large language models under imperfect guidance. |
| Outcome: | The proposed benchmark compared LLM agents in navigation, cooking, and gaming in a variety of environments with auxiliary guidance and noisy or underspecified instructions extracted from demonstrations. |
Plugging Schema Graph into Multi-Table QA: A Human-Guided Framework for Reducing LLM Reliance (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods based on semantic similarity work well only on simplified datasets . Existing approaches based only on semantic similarities struggle to handle complex tables . |
| Approach: | They propose a graph-based framework that leverages human-curated relational knowledge to explicitly encode schema links and join paths. |
| Outcome: | The proposed framework leverages human-curated relational knowledge to encode schema links and join paths. |
SEK: Self-Explained Keywords Empower Large Language Models for Code Generation (2025.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) have achieved impressive performance in code generation. |
| Approach: | They propose a technique that extracts and explicates the key terms in the problem description with the LLM itself. |
| Outcome: | The proposed technique improves the Pass@1 of DeepSeek-Coder-V2-Instruct from 85.4% to 93.3% on the humaneval benchmark. |
Progra: Progress-Aware Reinforcement Learning for Multi-Turn Function Calling (2026.findings-acl)
Copied to clipboard
Huacan Chai, Zijie Cao, Maolin Ran, Yingxuan Yang, Jianghao Lin, Xin Peng, Hairui Wang, Renjie Ding, Ziyu Wan, Muning Wen, Weiwen Liu, Weinan Zhang, Fei Huang, Ying Wen
| Challenge: | Existing methods for multi-turn function calling are limited by redundancy and lack explicit integration of progress awareness into training. |
| Approach: | They propose a framework that explicitly integrates progress awareness into LLM training for multi-turn function calling. |
| Outcome: | Empirical results show that Progra outperforms existing methods on two public benchmarks. |
Learning to Judge: LLMs Designing and Applying Evaluation Rubrics (2026.findings-eacl)
Copied to clipboard
| Challenge: | Large language models are increasingly used as evaluators for natural language generation . human rubrics are often static and misaligned with how models internally represent language quality. |
| Approach: | They propose to use large language models to generate interpretable and task-aware evaluation dimensions and apply them within models. |
| Outcome: | The proposed model improves the semantic coherence and scoring reliability of LLM-defined criteria and their alignment with human criteria. |
Incentivizing Strong Reasoning from Weak Supervision (2026.eacl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated impressive performance on reasoning-intensive tasks, but enhancing their reasoning abilities typically relies on expensive high-quality demonstrations and reinforcement learning. |
| Approach: | They propose to incentivize reasoning abilities of large language models without expensive demonstrations and reinforcement learning. |
| Outcome: | The proposed model can recover 94% of the gains of expensive RL at a fraction of the cost. |
Learning to Search Effective Example Sequences for In-Context Learning (2025.findings-naacl)
Copied to clipboard
| Challenge: | Existing methods address these factors in isolation, overlooking their interdependencies. Existing approaches focus on sequence selection, while focusing on the sequence of examples. |
| Approach: | They propose a method that considers key factors involved in sequence selection and incrementally builds the sequence. |
| Outcome: | Experiments across various datasets and language models show that the proposed method significantly reduces the search space and improves performance. |
EvoPrompt: Evolving Prompts for Enhanced Zero-Shot Named Entity Recognition with Large Language Models (2025.coling-main)
Copied to clipboard
| Challenge: | Named Entity Recognition (NER) is a low-resource task that requires supervised learning, but practical scenarios lack annotated data. |
| Approach: | They propose an Evolving Prompts framework that guides the model to better address these issues through continuous prompt refinement. |
| Outcome: | The proposed framework shows consistent performance improvements on four benchmarks. |
Effective Large Language Model Adaptation for Improved Grounding and Citation Generation (2024.naacl-long)
Copied to clipboard
| Challenge: | Large language models generate "hallucinated" answers that are not factual . despite their widespread adoption, they can generate plausiblesounding but nonfactual information. |
| Approach: | They propose a framework that tunes large language models to self-ground claims and provide citations to retrieved documents. |
| Outcome: | The proposed framework generates superior grounded responses with more accurate citations compared to prompting-based approaches and post-hoc citing-based methods. |
Exploring Large Language Models for Multi-Modal Out-of-Distribution Detection (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Out-of-distribution (OOD) detection is essential for reliable and trustworthy machine learning. |
| Approach: | They propose to apply world knowledge to enhance OOD detection performance through selective generation from large language models (LLMs) they propose to extract visual objects from each image to fully capitalize on the aforementioned world knowledge. |
| Outcome: | The proposed method outperforms the state-of-the-art on visual OOD detection on in-distribution (ID) samples. |
Beyond Under-Alignment: Atomic Preference Enhanced Factuality Tuning for Large Language Models (2025.findings-naacl)
Copied to clipboard
| Challenge: | Existing work evaluates the factuality of large language models on in-domain (ID) datasets and the factuality on out-of-domain datasets. |
| Approach: | They propose a framework that enhances model’s awareness of factuality at the granularity of individual facts and propose 'Atomic Preference Enhanced Factuality Tuning' this framework enhances the model’ s awareness and accuracy of factual information at the level of individual factual facts. |
| Outcome: | The proposed framework improves model performance by an average of on ID and OOD datasets, which is highly effective. |
SafeSwitch: Steering Unsafe LLM Behavior via Internal Activation Signals (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing safety mechanisms for large language models (LLMs) are inadequate to fully leverage their internal cognitive processes. |
| Approach: | They propose a framework that regulates unsafe outputs by utilizing the prober-based internal state monitor that actively detects harmful intentions. |
| Outcome: | The proposed framework reduces harmful outputs by approximately 80% while maintaining strong utility. |
A Survey of Confidence Estimation and Calibration in Large Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated impressive capabilities across a wide range of tasks in various domains, but they can be unreliable due to factual errors in their generations. |
| Approach: | They summarize recent advances in LLM confidence estimation and calibration and outline their main lessons learned. |
| Outcome: | The proposed methods can be used to assess the reliability of models and to calibrate them across tasks. |
Large Language Model-Enhanced Multi-Armed Bandits (2026.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) have been used to sequential decision-making tasks like multi-armed bandits where an LLM is tasked with selecting arms in each iteration is often suboptimal. |
| Approach: | They propose to combine MAB and LLMs to leverage the in-context learning capability of LLM for reward prediction. |
| Outcome: | The proposed approach outperforms LLM-based direct arm selection on synthetic tasks where only preference feedback between arm pairs is available. |
TopicAttack: An Indirect Prompt Injection Attack via Topic Transition (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent attacks have demonstrated potential, but their abrupt instruction injection often undermines their effectiveness. |
| Approach: | They propose a method that prompts the LLM to generate a fabricated conversational transition prompt that gradually shifts the topic toward the injected instruction. |
| Outcome: | The proposed method achieves state-of-the-art performance with an attack success rate (ASR) over 90% in most cases, even when various defense methods are applied. |
Teaching Small Language Models to Learn Logic through Meta-Learning (2026.eacl-long)
Copied to clipboard
| Challenge: | Large language models are increasingly evaluated on reasoning tasks, yet their logical abilities remain contested. |
| Approach: | They propose to apply few-shot meta-learning to large language models' reasoning domain to enable them to acquire abstract inference patterns that generalize to novel structures. |
| Outcome: | The proposed model outperforms GPT-4o and o3-mini on a syllogistic reasoning task. |
FinMaster: A Holistic Benchmark for Full-Pipeline Financial Management with Large Language Models (2026.findings-acl)
Copied to clipboard
Junzhe Jiang, Chang Yang, Aixin Cui, Sihan Jin, Yujing Zhang, Yilin Xiao, Ruiyu Wang, Bo Li, Xiao Huang, Danny Dongning Sun, Xinrun Wang
| Challenge: | Existing benchmarks lack domain-specific data, realistic workflow-level task design, and standardized workflow- level evaluation. |
| Approach: | a new benchmark evaluates large language models on financial management workflows . the global financial services market is projected to grow to $37 trillion by 2027 . |
| Outcome: | a new benchmark for large language models on financial management workflows reveals critical capability gaps . accuracy drops from 90% on basic tasks to 40% on complex scenarios requiring multi-step reasoning . the global financial services market reached $25.8 trillion in 2022 and is projected to grow to $37 trillion by 2027 . |
Noise, Adaptation, and Strategy: Assessing LLM Fidelity in Decision-Making (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) are increasingly used for social science simulations . however, most evaluations focus on task optimality rather than variability and adaptation characteristic of human decision-making. |
| Approach: | They propose a process-oriented evaluation framework with progressive interventions to evaluate two economics tasks using large language models. |
| Outcome: | The proposed evaluation framework targets two economic tasks with progressive interventions. |
An empirical study of validating synthetic data for formula generation (2025.findings-naacl)
Copied to clipboard
Usneek Singh, José Cambronero, Sumit Gulwani, Aditya Kanade, Anirudh Khatry, Vu Le, Mukul Singh, Gust Verbruggen
| Challenge: | Large language models (LLMs) can be leveraged to help write formulas in spreadsheets, but formula data resources are scarce, limiting the ability to fine-tune them. |
| Approach: | They validate a corpus of formulas with a model to generate synthetic natural language utterances for fine-tuning. |
| Outcome: | The proposed model generates synthetic natural language utterances with a model that is accurate enough to fine-tune them. |
Large Language Models for Persian-English Idiom Translation (2025.naacl-long)
Copied to clipboard
| Challenge: | Large language models have shown superior capabilities in translating figurative language compared to neural machine translation systems. |
| Approach: | They evaluate LLMs, NMTs and their combinations using PersianIdioms datasets . they find that automatic evaluation methods like BLEU and BERTScore are effective . |
| Outcome: | The proposed model performs better in both directions than other models. |
EnvScaler: Scaling Tool-Interactive Environments for LLM Agent via Programmatic Synthesis (2026.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) are expected to be trained to act as agents in various real-world environments, but access to real systems is often restricted and manually built sandboxes are hard to scale. |
| Approach: | They propose an automated framework for scalable tool-interaction environments via programmatic synthesis that synthesizes 191 environments and about 7K scenarios and applies them to Supervised Fine-Tuning and Reinforcement Learning for Qwen3 series models. |
| Outcome: | The proposed framework significantly improves LLMs’ ability to solve tasks in complex environments involving multi-turn, multi-tool interactions. |
SPASM: Stable Persona-driven Agent Simulation for Multi-turn Dialogue Generation (2026.findings-acl)
Copied to clipboard
| Challenge: | Large language models are increasingly deployed in multi-turn settings such as tutoring, support, and counseling where reliability depends on preserving consistent roles, personas, and goals across long horizons. |
| Approach: | They propose a framework that decomposes LLM–LLM conversations into a modular, stability-first framework that allows for a stable persona-driven agent simulation for multi-turn dialogue generation. |
| Outcome: | The proposed framework decomposes the LLM-based model into four main components: persona creation, plausibility validation, and natural-language persona crafting. |
Teaching Models to Balance Resisting and Accepting Persuasion (2025.naacl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) are susceptible to persuasion, which can pose risks when faced with an adversarial interlocutor. |
| Approach: | They propose a method to balance positive and negative persuasion by using recursive dialogue trees to train models to accept persulasion. |
| Outcome: | The proposed model-based training improves resistance to misinformation and resilience to being challenged while also resulting in the best overall performance on multi-agent debates across two domains. |
HD-Eval: Aligning Large Language Model Evaluators Through Hierarchical Criteria Decomposition (2024.acl-long)
Copied to clipboard
Yuxuan Liu, Tianchi Yang, Shaohan Huang, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, Qi Zhang
| Challenge: | Large language models (LLMs) are a promising alternative to expensive human evaluations. |
| Approach: | They propose a framework that iteratively aligns LLM-based evaluators with human preference . they decompose a given evaluation task into finer-grained criteria . |
| Outcome: | The proposed framework iteratively aligns LLM-based evaluators with human preference . it decomposes a given evaluation task into finer-grained criteria . the framework is efficient to train and more explainable than relying solely on prompts . |
Modeling Human Subjectivity in LLMs Using Explicit and Implicit Human Factors in Personas (2024.findings-emnlp)
Copied to clipboard
Salvatore Giorgi, Tingting Liu, Ankit Aich, Kelsey Isman, Garrick Sherman, Zachary Fried, João Sedoc, Lyle Ungar, Brenda Curtis
| Challenge: | Large language models (LLMs) are increasingly being used in human-centered social scientific tasks, such as data annotation, synthetic data creation, and engaging in dialog. |
| Approach: | They propose to prompt LLMs with human-like personas and ask them to answer as if they were a specific human, either explicitly, with exact demographics, political beliefs, and lived experiences, or implicitly via names prevalent in specific populations. |
| Outcome: | The proposed model is based on explicit, explicit, and implicit personas, and fails to show implicit biases. |
Steering Language Models in Multi-Token Generation: A Case Study on Tense and Aspect (2025.emnlp-main)
Copied to clipboard
| Challenge: | Prior work has focused largely on binary grammatical contrasts, but how do they encode their syntactic knowledge internally? |
| Approach: | They propose to use a multidimensional hierarchical grammar phenomenon to identify distinct, orthogonal directions in residual space to demonstrate causal control over both grammatical features. |
| Outcome: | The proposed model can encode tense and aspect in human-like ways, but effective steering during generation is sensitive to multiple factors and requires manual tuning or automated optimization. |
Reason to Rote: Rethinking Memorization in Reasoning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models readily memorize arbitrary training instances, such as label noise . however, such memorization does not affect generalizable reasoning abilities . |
| Approach: | They investigate how large language models memorize label noise and why it affects generalizability. |
| Outcome: | The proposed model performs well on reasoning tasks even when memorized labels are missing . the proposed model is able to generalize to correctly answer "87+19=106" |
Query Refinement Prompts for Closed-Book Long-Form QA (2023.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) can answer questions and produce long-form texts, but the latter is difficult to evaluate since they are subjective in nature. |
| Approach: | They propose query refinement prompts that encourage LLMs to express multifacetedness and generate long-form answers covering multiple facets of the question. |
| Outcome: | The proposed model outperforms fully finetuned models in the closed-book setting and retrieve-then-generate open-book models. |
Contrastive Learning for Task-Independent SpeechLLM-Pretraining (2025.findings-acl)
Copied to clipboard
| Challenge: | Large language models excel in speech processing tasks but their reliance on written text limits their application in real-world scenarios. |
| Approach: | They propose a task-independent speech pretraining stage and task-specific fine-tuning stage to adapt LLMs to speech processing tasks. |
| Outcome: | The proposed model outperforms models specialized on speech translation and question answering while being trained on 10% of the task-specific data. |
Developmentally-plausible Working Memory Shapes a Critical Period for Language Acquisition (2025.acl-long)
Copied to clipboard
| Challenge: | Large language models possess general linguistic abilities comparable to humans but their efficiency in language acquisition remains far inferior. |
| Approach: | They propose a method that initially constrains working memory during the early stages of training and gradually relaxes this constraint as learning progresses. |
| Outcome: | The proposed method outperforms conventional methods without memory constraints or with static memory constraints. |
StoryMI: Steerable Multi-Agent Therapeutic Dialogue Generation (2026.findings-acl)
Copied to clipboard
| Challenge: | Motivational interviewing (MI) is a directive, client-centered counseling approach for eliciting clients' motivation for behavioral change. |
| Approach: | They propose a multi-LLM agent framework for controllable MI dialogue generation . therapist and client agents generate MI-coded utterances guided by MI codes . |
| Outcome: | The proposed framework can generate fluent dialogues with minimal intervention time and a high level of evaluation. |
Measuring memorization in language models via probabilistic extraction (2025.naacl-long)
Copied to clipboard
Jamie Hayes, Marika Swanberg, Harsh Chaudhari, Itay Yona, Ilia Shumailov, Milad Nasr, Christopher A. Choquette-Choo, Katherine Lee, A. Feder Cooper
| Challenge: | Large language models (LLMs) are susceptible to memorizing training data, raising concerns about the potential extraction of sensitive information at generation time. |
| Approach: | They propose a method that splits training example into prefix and suffix, prompts LLM with suffix and deems it extractable if it generates the suffix using greedy sampling. |
| Outcome: | The proposed method is unreliable because it does not account for non-determinism in more realistic sampling schemes. |
Enhancing LLM Knowledge Learning through Generalization (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Continued pre-training on paraphrased data has shown empirical promise for enhancing knowledge acquisition, but this approach is costly and unreliable as it relies on external models or manual effort for rewriting. |
| Approach: | They propose formatting-based data augmentation which diversifies documents conveying the same knowledge by altering document formats rather than their content. |
| Outcome: | The proposed methods improve generalization to diverse paraphrased contexts and enhance pre-training and instruction tuning. |
Two Directions for Clinical Data Generation with Large Language Models: Data-to-Label and Label-to-Data (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) can generate natural language texts for various domains and tasks, but their potential for clinical text mining is under-explored. |
| Approach: | They propose a pragmatic taxonomy for AD sign and symptom progression based on expert knowledge and train a system to detect AD-related signs and symptoms from EHRs. |
| Outcome: | The proposed taxonomy outperforms existing methods using only the gold dataset and silver datasets. |
MASS-RAG: Multi-Agent Synthesis Retrieval-Augmented Generation (2026.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) are widely used in retrieval-augmented generation (RAG) when retrieved contexts are noisy, incomplete, or heterogeneous, a single generation process often struggles to reconcile evidence effectively. |
| Approach: | They propose a multi-agent synthesis approach to retrieval-augmented generation that structures evidence processing into multiple role-specialized agents. |
| Outcome: | Experiments on four benchmarks show that MASS-RAG consistently improves performance over strong RAG baselines. |
Characterizing Large Language Models as Rationalizers of Knowledge-intensive Tasks (2024.findings-acl)
Copied to clipboard
| Challenge: | Large language models generate fluent text with minimal task-specific supervision, but their ability to generate rationales for knowledge-intensive tasks (KITs) remains under-explored. |
| Approach: | They propose to generate retrieval-augmented rationalization of KIT model predictions via external knowledge guidance within a few-shot setting. |
| Outcome: | The proposed rationales were compared with crowd-sourced rationale models on factuality, sufficiency, and convincingness. |
Scaling Laws for Code: Every Programming Language Matters (2026.findings-acl)
Copied to clipboard
Jian Yang, Shuyue Guo, Linzheng Chai, Wei Zhang, Aishan Liu, Chuan Hao, Zhoujun Li, Xin Zhao, Xianglong Liu, Weifeng Lv, Bryan Dai
| Challenge: | Existing studies focus on language-agnostic settings, neglecting the inherently multilingual nature of modern software development. |
| Approach: | They propose a proportion-dependent scaling law that prioritizes high-utility languages . they propose PLs to have varying effects during pre-training that affect model performance . |
| Outcome: | The proposed scaling law is based on 1000+ experiments across multiple languages and models. |
Valid Survey Simulations with Limited Human Data: The Roles of Prompting, Fine-Tuning, and Rectification (2026.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) are a cost-effective and time-consuming way to capture public opinion and behavior, but their outputs are often biased and yield invalid estimates. |
| Approach: | They propose to use large language models to generate survey responses and rectification methods that debias population estimates to find out how human responses are best allocated between them. |
| Outcome: | The proposed methods reduce bias below 5% and increase sample size by up to 14% under a fixed budget. |
Focus on Your Question! Interpreting and Mitigating Toxic CoT Problems in Commonsense Reasoning (2024.acl-long)
Copied to clipboard
| Challenge: | Large language models exhibit high-level commonsense reasoning abilities, especially with enhancement methods like Chain-of-Thought (CoT). |
| Approach: | They propose a chain-of-thought-like method to elicit models' potential abilities to generate rationales and answers that are based on attribution tracing and causal tracers to probe the internal working mechanism of the LLM. |
| Outcome: | The proposed method eliminates Toxic CoT problems and improves the model’s overall commonsense reasoning performance by 5.5%. |
Perspective Transition of Large Language Models for Solving Subjective Tasks (2025.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) have revolutionized the field of natural language processing . performance of LLMs on subjective tasks is limited, authors say . |
| Approach: | They propose a method that allows LLMs to select between direct, role, and third-person perspectives for best way to solve corresponding subjective problem. |
| Outcome: | The proposed method outperforms widely used single fixed perspective based methods on 12 subjective tasks. |
Towards Consistent Natural-Language Explanations via Explanation-Consistency Finetuning (2025.coling-main)
Copied to clipboard
| Challenge: | Large language models generate convincing, fluent explanations, but they often generate inconsistent explanations on different inputs. |
| Approach: | They propose a method that adapts large language models to generate more consistent explanations on related examples. |
| Outcome: | The proposed method yields a 10.0% relative explanation consistency improvement across a variety of question-answering datasets and generalizes to 7 out-of-distribution datasets not seen during finetuning (+4.5%) |
CartesianMoE: Boosting Knowledge Sharing among Experts via Cartesian Product Routing in Mixture-of-Experts (2025.naacl-long)
Copied to clipboard
Zhenpeng Su, Xing W, Zijia Lin, Yizhe Xiong, Minxuan Lv, Guangyuan Ma, Hui Chen, Songlin Hu, Guiguang Ding
| Challenge: | Large language models (LLMs) have been attracting much attention due to their impressive performance in all kinds of downstream tasks. |
| Approach: | They propose a mix-of-experts model that allows the model size to grow without raising training costs. |
| Outcome: | The proposed model outperforms existing models in perplexity and robustness tests. |
Qorǵau: Evaluating Safety in Kazakh-Russian Bilingual Contexts (2025.findings-acl)
Copied to clipboard
Maiya Goloburda, Nurkhan Laiyk, Diana Turmakhan, Yuxia Wang, Mukhammed Togmanov, Jonibek Mansurov, Askhat Sametov, Nurdaulet Mukhituly, Minghan Wang, Daniil Orel, Zain Muhammad Mujahid, Fajri Koto, Timothy Baldwin, Preslav Nakov
| Challenge: | Large language models (LLMs) have the potential to generate harmful content, posing risks to users. |
| Approach: | They propose a dataset specifically designed for safety evaluation in Kazakh and Russian . they use a bilingual context in Kazakhstan where both Kazakh (a low-resource language) and Russian (a high-resourced language) |
| Outcome: | The proposed dataset is designed for safety evaluation in Kazakh and Russian . it shows that both multilingual and language-specific LLMs perform better than others . |
Reasoning with Language Model is Planning with World Model (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) have shown remarkable reasoning capabilities, particularly with Chain-of-Thought-style prompts. |
| Approach: | They propose a framework that repurposes the LLM as both a world model and a reasoning agent and incorporates a principled planning algorithm (based on Monte Carlo Tree Search) |
| Outcome: | The proposed framework repurposes the LLM as both a world model and a reasoning agent and incorporates a principled planning algorithm (based on Monte Carlo Tree Search) it achieves optimum balance between exploration and exploitation, while achieving high-reward reasoning paths efficiently. |
DB-LLM: Accurate Dual-Binarization for Efficient LLMs (2024.findings-acl)
Copied to clipboard
Hong Chen, Chengtao Lv, Liang Ding, Haotong Qin, Xiabin Zhou, Yifu Ding, Xuebo Liu, Min Zhang, Jinyang Guo, Xianglong Liu, Dacheng Tao
| Challenge: | Existing methods for ultra-low bit quantization cause severe accuracy drops . a novel Dual-Binarization method is proposed for efficient Large Language Models . |
| Approach: | They propose a Dual-Binarization method that takes 2-bit-width and binarization into account . they propose DB-LLM, which uses a 2-bit binarized weighted model to represent weights efficiently . |
| Outcome: | The proposed method surpasses the current State-of-the-Art in ultra-low bit quantization and achieves 20% reduction in computational consumption compared to the SOTA method under the same bit-width. |
What do Large Language Models Learn beyond Language? (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Pretraining on text confers models with useful ‘inductive biases’ for non-linguistic reasoning. |
| Approach: | They investigate whether pre-training on text confers these models with helpful ‘inductive biases’ for non-linguistic reasoning. |
| Outcome: | The proposed models outperform non-pretrained models on 19 non-linguistic tasks and show that they retain inductive biases even when training on multi-lingual text and computer code. |
Impact of Co-occurrence on Factual Knowledge of Large Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) often make factually incorrect responses despite their success in various applications. |
| Approach: | They propose to fine tune large language models to mitigate the co-occurrence bias by filtering out biased samples with high subject-object co-occurring counts. |
| Outcome: | The proposed model scales up to debiased datasets to mitigate the co-occurrence bias, but is not effective in recalling rare facts unseen during finetuning. |
Unveiling Environmental Impacts of Large Language Model Serving: A Functional Unit View (2025.acl-long)
Copied to clipboard
| Challenge: | Existing studies benchmark carbon emissions but lack a standardized basis for comparison across different model configurations. |
| Approach: | They propose a functional unit (FU) as a standardized basis for comparison across different model configurations and introduce FUEL, the first FU-based framework for evaluating LLM serving’s environmental impact. |
| Outcome: | The proposed framework uncovers key insights and trade-offs in reducing carbon emissions by optimizing model size, quantization strategy, and hardware choice, paving the way for more sustainable LLM serving. |
When Long Helps Short: How Context Length in Supervised Fine-tuning Affects Behavior of Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) have achieved impressive performance across NLP tasks. |
| Approach: | They propose to use long-context SFT to improve short-contemporary performance . they also decouple and analyze two key components, Multi-Head Attention and Feed-Forward Network . |
| Outcome: | The proposed model improves short-context performance, contrary to pretraining. |
Bit-Flip Error Resilience in LLMs: A Comprehensive Analysis and Defense Framework (2025.emnlp-main)
Copied to clipboard
Yuhang Chen, Zhen Tan, Ajay Kumar Jaiswal, Huaizhi Qu, Xinyu Zhao, Qi Lin, Yu Cheng, Andrew Kwong, Zhichao Cao, Tianlong Chen
| Challenge: | Bit-flip errors (BFEs) are hardware faults where individual bits in memory or processing units are unintentionally flipped. |
| Approach: | They propose a novel defense strategy to mitigate bit-flip errors (BFEs) they propose bfe protection and a self-correction mechanism to minimize performance degradation . |
| Outcome: | The proposed defense strategy minimizes performance degradation while significantly improving robustness against BFEs. |
SATER: A Self-Aware and Token-Efficient Approach to Routing and Cascading (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) demonstrate remarkable performance across diverse tasks, yet their effectiveness often depends on costly commercial APIs or cloud services. |
| Approach: | They propose a dual-mode compatible approach that fine-tunes models through shortest-response preference optimization and a confidence-aware rejection mechanism. |
| Outcome: | The proposed approach reduces redundant outputs and response times while reducing computational costs by over 50% and cascade latency by over 80%. |
DSG-MCTS: A Dynamic Strategy-Guided Monte Carlo Tree Search for Diversified Reasoning in Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) have shown strong potential in complex reasoning tasks, but their performance often degrades, resulting in hallucinations, errors, and logical inconsistencies. |
| Approach: | They propose a framework that integrates multiple reasoning strategies to expand the reasoning space and a dynamic strategy selection mechanism that adapts to the task context. |
| Outcome: | The proposed framework outperforms existing state-of-the-art methods on a set of reasoning benchmarks. |
Latent Inter-User Difference Modeling for LLM Personalization (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) are increasingly integrated into users’ daily lives, leading to a growing demand for personalized outputs. |
| Approach: | They propose a framework that models inter-user differences in the latent space instead of relying on language-based prompts. |
| Outcome: | The proposed framework outperforms baseline methods on personalized review generation. |
ARise: Towards Knowledge-Augmented Reasoning via Risk-Adaptive Search (2025.acl-long)
Copied to clipboard
Yize Zhang, Tianshu Wang, Sirui Chen, Kun Wang, Xingyu Zeng, Hongyu Lin, Xianpei Han, Le Sun, Chaochao Lu
| Challenge: | Large language models (LLMs) have impressive capabilities but their application in open-ended, knowledge-intensive, complex reasoning scenarios is limited. |
| Approach: | They propose a framework that integrates risk assessment of intermediate reasoning states with dynamic retrieval-augmented generation within a Monte Carlo tree search paradigm. |
| Outcome: | The proposed framework outperforms the state-of-the-art KAR methods by up to 23.10% and the latest RAG-equipped large reasoning models by upto 25.37%. |
The LLM Language Network: A Neuroscientific Approach for Identifying Causally Task-Relevant Units (2025.naacl-long)
Copied to clipboard
| Challenge: | Recent advances in large language models (LLMs) have revealed their potential to perform far more than language processing tasks, showcasing abilities in reasoning and problem-solving. |
| Approach: | They identify language-selective units within 18 popular LLMs using the same localization approach that is used in neuroscience. |
| Outcome: | The proposed method shows that language-selective units are more aligned to brain recordings from the human language system than random units. |
RAG or Learning? Understanding the Limits of LLM Adaptation under Continuous Knowledge Drift in the Real World (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to update or supplement large language models struggle under continuous knowledge drift. |
| Approach: | They propose a dynamic event benchmark and time-aware retrieval baseline that captures how knowledge evolves over time. |
| Outcome: | The proposed method enables systematic evaluation of model adaptation under continuous knowledge drift. |
Improving Factuality with Explicit Working Memory (2025.acl-long)
Copied to clipboard
Mingda Chen, Yang Li, Karthik Padthe, Rulin Shao, Alicia Yi Sun, Luke Zettlemoyer, Gargi Ghosh, Wen-tau Yih
| Challenge: | Large language models can generate factually inaccurate content, a problem known as hallucination. |
| Approach: | They propose an approach that integrates a working memory that receives feedback from external resources. |
| Outcome: | The proposed method outperforms baselines on four fact-seeking datasets and increases the factuality metric by 2 to 6 points absolute. |
On the Vulnerability of Safety Alignment in Open-Access LLMs (2024.findings-acl)
Copied to clipboard
Jingwei Yi, Rui Ye, Qisi Chen, Bin Zhu, Siheng Chen, Defu Lian, Guangzhong Sun, Xing Xie, Fangzhao Wu
| Challenge: | Large language models (LLMs) are susceptible to malicious exploitation, but are often rejected and limited harmfulness is limited. |
| Approach: | They propose two types of reverse alignment techniques: reverse supervised fine-tuning (RSFT) and reverse preference optimization (RPO). |
| Outcome: | The proposed methods can significantly enhance the success rate and harmfulness of jailbreak attacks, but they face high rejection rates and limited harmfulness. |
Say What You Mean! Large Language Models Speak Too Positively about Negative Commonsense Knowledge (2023.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) have been studied for their ability to store and utilize positive knowledge. |
| Approach: | They propose to use a constrained keywords-to-sentence generation task and a Boolean question answering task to probe large language models on negative commonsense knowledge. |
| Outcome: | The proposed tasks show that LLMs fail to generate valid sentences grounded in negative commonsense knowledge, yet they can correctly answer yes-or-no questions. |
“You Are An Expert Linguistic Annotator”: Limits of LLMs as Analyzers of Abstract Meaning Representation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) demonstrate proficiency and fluency in the use of language, but do they have the linguistic knowledge to serve as an expert linguistic annotator? |
| Approach: | They examine the successes and limitations of large language models using the Abstract Meaning Representation (AMR) parsing formalism. |
| Outcome: | The proposed models can reproduce the basic format of AMR, as well as some core event, argument, and modifier structure, but they have virtually no fully accurate parses. |
Quantifying the Persona Effect in LLM Simulations (2024.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) have shown remarkable promise in simulating human language and behavior. |
| Approach: | They investigate how integrating persona variables—demographic, social, and behavioral factors—impacts LLMs’ ability to simulate diverse perspectives. |
| Outcome: | The proposed model improves on a zero-shot model with persona prompting. |
GuideBench: Benchmarking Domain-Oriented Guideline Following for LLM Agents (2025.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) have been widely deployed as autonomous agents capable of following user instructions and making decisions in real-world applications. |
| Approach: | They propose a benchmark to evaluate LLMs' ability to follow domain-oriented guidelines . they evaluate Lms on three critical aspects: adherence to diverse rules, robustness to rule updates . |
| Outcome: | The proposed benchmark evaluates LLMs on three critical aspects: adherence to diverse rules, robustness to rule updates, and alignment with human preferences. |
Stimulate the Critical Thinking of LLMs via Debiasing Discussion (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies show that large language models (LLMs) are often prone to stance homogeneity and human preference biases when faced with conflicting perspectives. |
| Approach: | They propose a novel two-stage training framework to address stance homogeneity bias and human preference bias by generating multi-model discussion datasets and optimizing reinforcement learning from human feedback to align with discussion correctness. |
| Outcome: | The proposed framework reduces stance homogeneity bias and human preference bias and improves generalization capabilities on non-discussion scenarios and out-of-domain datasets. |
Analyzing and Improving Coherence of Large Language Models in Question Answering (2025.naacl-long)
Copied to clipboard
| Challenge: | Large language models suffer from instability or lack of coherence when receiving diverse input variations. |
| Approach: | They analyze the behavior of large language models when dealing with multiple lexical variations of the same info-seeking questions. |
| Outcome: | The proposed model generates equivalent outputs when receiving diverse input variations. |
Make-A-Voice: Revisiting Voice Large Language Models as Scalable Multilingual and Multitask Learners (2024.acl-long)
Copied to clipboard
Rongjie Huang, Chunlei Zhang, Yongqi Wang, Dongchao Yang, Jinchuan Tian, Zhenhui Ye, Luping Liu, Zehan Wang, Ziyue Jiang, Xuankai Chang, Jiatong Shi, Chao Weng, Zhou Zhao, Dong Yu
| Challenge: | Large language models (LLMs) have been used for general-purpose interfaces across multiple tasks and languages. |
| Approach: | They propose to use large language models as a general-purpose interface across multiple tasks and languages. |
| Outcome: | The proposed model performs better on 200K hours of 6-language data for voice generation applications. |
Evolving Knowledge Distillation with Large Language Models and Active Learning (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing studies have focused on the direct use of large language models for text generation and labeling, without fully exploring their potential to comprehend the target task and acquire valuable knowledge. |
| Approach: | They propose to distill the knowledge of large language models into smaller models by generating annotated data. |
| Outcome: | The proposed method improves the performance of small domain models while enhancing the ability of large language models. |
Extending LLMs to New Languages: A Case Study of Llama and Persian Adaptation (2025.coling-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) are mainly trained on English data and struggle with low-resource languages. |
| Approach: | They propose to add a new language to Llama to improve classification accuracy for Persian tasks by aligning representations through bilingual pretraining and instruction datasets. |
| Outcome: | The proposed model performs on generation and classification tasks with no adverse impact and sometimes even improvements on English tasks. |
LEMMA: Learning from Errors for MatheMatical Advancement in LLMs (2025.findings-acl)
Copied to clipboard
Zhuoshi Pan, Yu Li, Honglin Lin, Qizhi Pei, Zinan Tang, Wei Wu, Chenlin Ming, H. Vicky Zhao, Conghui He, Lijun Wu
| Challenge: | Existing approaches focus on improving the quality of correct training data, neglecting the value contained in error data, thereby hindering the model’s reflective ability. |
| Approach: | They propose to improve LLM's reasoning ability by learning from error data and a grounded mistake augmentation method to collect representative errors. |
| Outcome: | The proposed model achieves significant performance improvements over other strong models with less than 90k data. |
Balancing Classification and Calibration Performance in Decision-Making LLMs via Calibration Aware Reinforcement Learning (2026.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) are increasingly deployed in decision-making tasks where accuracy and reliable confidence estimates are essential. |
| Approach: | They propose a calibration-aware reinforcement learning formulation that directly adjusts decision-token probabilities. |
| Outcome: | The proposed model preserves RLVR’s accuracy level while mitigating overconfidence, reducing ECE scores up to 9 points. |
Rethinking Parameter Sharing for LLM Fine-Tuning with Multiple LoRAs (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for large language models use parameter-efficient techniques such as Low-Rank Adaptation (LoRA) prior studies suggest that the inner A matrices are highly similar during training and therefore suitable for sharing. |
| Approach: | They propose an asymmetric multi-LoRA design with multiple A matrices and a single shared B in multi-task fine-tuning. |
| Outcome: | The proposed methods achieve more balanced performance across tasks with comparable or superior average accuracy relative to existing methods. |
Towards Advanced Mathematical Reasoning for LLMs via First-Order Logic Theorem Proving (2025.emnlp-main)
Copied to clipboard
Chuxue Cao, Mengze Li, Juntao Dai, Jinluan Yang, Zijian Zhao, Shengyu Zhang, Weijie Shi, Chengzhong Liu, Sirui Han, Yike Guo
| Challenge: | Large language models (LLMs) have shown promising first-order logic (FOL) reasoning capabilities with applications in various areas, but their effectiveness in complex mathematical reasoning involving multi-step FOL deductions remains under-explored. |
| Approach: | They propose a self-adaptive solution that enhances the Diversity and REAsonability of LLMs’ generation strategies by introducing an Axiom-Driven Strategy Diversification mechanism and a Sub-Proposition Error Feedback to help LLM reflect on and correct their proofs. |
| Outcome: | The proposed model improves diversity and REAsonability of LLMs’ generation strategies by introducing an Axiom-Driven Strategy Diversification mechanism and a Sub-Proposition Error Feedback to help LLM reflect on and correct proofs. |
Revealing and Mitigating the Local Pattern Shortcuts of Mamba (2025.findings-acl)
Copied to clipboard
| Challenge: | Recent studies show that Mamba excels in tasks that involve localized key information but faces challenges with tasks that require handling distributed key information. |
| Approach: | They propose to introduce a global gate module into Mamba to address this problem by adding 4M extra parameters to the model. |
| Outcome: | The proposed model outperforms attention-based models on synthetic and synthetic tasks with only 4M extra parameters. |
PerCul: A Story-Driven Cultural Evaluation of LLMs in Persian (2025.naacl-long)
Copied to clipboard
Erfan Moosavi Monazzah, Vahid Rahimzadeh, Yadollah Yaghoobzadeh, Azadeh Shakery, Mohammad Taher Pilehvar
| Challenge: | Large language models predominantly reflect Western cultures due to the dominance of English-centric training data. |
| Approach: | They propose a dataset to assess the sensitivity of LLMs to Persian culture. |
| Outcome: | The proposed model shows a 11.3% gap between best closed-source model and layperson baseline while the gap increases to 21.3% by using the best open-weight model. |
Rationale-Guided Retrieval Augmented Generation for Medical Question Answering (2025.naacl-long)
Copied to clipboard
Jiwoong Sohn, Yein Park, Chanwoong Yoon, Sihyeon Park, Hyeon Hwang, Mujeen Sung, Hyunjae Kim, Jaewoo Kang
| Challenge: | Large language models (LLMs) struggle with hallucinations and outdated knowledge. |
| Approach: | They propose a retrieval-augmented generation framework for enhancing the reliability of RAG in biomedical contexts. |
| Outcome: | The proposed framework outperforms the previous best medical RAG model by up to 5.6% across three medical question-answering benchmarks. |
Gendered Grammar or Ingrained Bias? Exploring Gender Bias in Icelandic Language Models (2024.lrec-main)
Copied to clipboard
| Challenge: | Large language models are trained on vast datasets and exhibit increased output quality in proportion to the amount of data that is used to train them. |
| Approach: | They explore whether language models mirror gender distributions within professions or exhibit biases tied to their grammatical genders. |
| Outcome: | The proposed model may reflect and amplify gender bias, racism, religious prejudice, and queerphobia in training data that may not always be recent. |
LightThinker: Thinking Step-by-Step Compression (2025.emnlp-main)
Copied to clipboard
Jintian Zhang, Yuqi Zhu, Mengshu Sun, Yujie Luo, Shuofei Qiao, Lun Du, Da Zheng, Huajun Chen, Ningyu Zhang
| Challenge: | Recent advances in Large Language Models have demonstrated their remarkable capabilities in complex reasoning tasks, but their efficiency is hindered by the substantial memory and computational costs associated with generating lengthy tokens. |
| Approach: | They propose a method that dynamically compresses verbose thought steps into compact representations and discards original reasoning chains. |
| Outcome: | The proposed method reduces peak memory usage and inference time while maintaining competitive accuracy. |
RoseRAG: Robust Retrieval-augmented Generation with Small-scale LLMs via Margin-aware Preference Optimization (2025.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) have impressive performance but face high computational costs and latency, limiting their deployment in resource-constrained settings. |
| Approach: | They propose a robust RAG framework for large language models via Margin-aware Preference Optimization to enhance the accuracy and reliability of SLMs. |
| Outcome: | The proposed framework surpasses state-of-the-art benchmarks on three open-domain question answering tasks. |
FastMem: Fast Memorization of Prompt Improves Context Awareness of Large Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models struggle with context awareness, leading to inaccuracies in tasks requiring faithful adherence to provided information. |
| Approach: | They propose a method to enhance LLMs' context awareness by updating only the last Feed-Forward Network module to maximize the likelihood of the prompt before inference . |
| Outcome: | The proposed method improves the accuracy of Llama 3-8B-Inst on the NQ-SWAP dataset from 59.1% to 71.6% and reduces the output structure failure rate of Qwen 1.5-4B-Chat from 34.9% to 25.5%. |
Self-Powered LLM Modality Expansion for Large Speech-Text Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large language models exhibit remarkable performance across diverse tasks . however, these methods require significant resource demands and tend to overfit specific tasks. |
| Approach: | They propose a self-powered LSM that leverages augmented automatic speech recognition data generated by the model itself for more effective instruction tuning. |
| Outcome: | The proposed model mitigates speech anchor bias and improves the fusion of speech and text modalities in large language models. |
CReTIHC: Designing Causal Reasoning Tasks about Temporal Interventions and Hallucinated Confoundings (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated impressive capabilities in natural language processing, but their ability to establish causal relationships remains challenging. |
| Approach: | They propose a novel dataset to test and enhance the causal reasoning abilities of large language models (LLMs) by integrating elements of verbal hallucinations and temporal interventions into existing causal inference datasets. |
| Outcome: | The proposed dataset is designed to test and enhance the causal reasoning abilities of large language models. |
Cross-Lingual Multi-Hop Knowledge Editing (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Prior work on knowledge editing in monolingual settings focused on a single language, but there are significant gaps in performance between the two settings. |
| Approach: | They propose a cross-lingual multi-hop knowledge editing paradigm for measuring and analyzing the performance of various SoTA knowledge editing techniques in a multilingual setup. |
| Outcome: | The proposed system improves on previous methods in a cross-lingual setting and in English. |
LLM Agents in Law: Taxonomy, Applications, and Challenges (2026.acl-long)
Copied to clipboard
Shuang Liu, Ruijia Zhang, Ruoyun Ma, Yujia Deng, Lanyi Zhu, Jiayu Li, Zelong Li, Zhibin Shen, Mengnan Du
| Challenge: | Large language models (LLMs) have improved the legal domain, but deployment of standalone models faces significant limitations regarding hallucination, outdated information, and verifiability. |
| Approach: | They present a survey of LLM agents for legal tasks and analyze their architectures . they analyze the transition from standard legal LLMs to legal agents . |
| Outcome: | The proposed architectures bridge the gap between technical capabilities and domain-specific needs. |
How Speculative Can Speculative Decoding Be? (2024.lrec-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) have a largely increased latency due to their ability to autoregressively model . speculative decoding is a technique that trades generation quality for speed . |
| Approach: | They propose to use a draft model to draft tokens autoregressively and then verify them in parallel. |
| Outcome: | The proposed model could draft tokens autoregressively and then verify them in parallel . the proposed model trades quality for speed and could fail in verification stage . |
Prediction-Augmented Generation for Automatic Diagnosis Tasks (2025.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) adopt autoregressive architecture, predicting the next word token based on the preceding context. |
| Approach: | They propose a method that integrates task-specific predictive models as external tools to improve model generation quality and accuracy. |
| Outcome: | The proposed method improves the generation quality and predictive accuracy of large language models in inference-driven tasks. |
The Path Not Taken: Duality in Reasoning about Program Execution (2026.acl-long)
Copied to clipboard
| Challenge: | Existing benchmarks focus on predicting program properties tied to specific inputs, resulting in a narrow view of dynamic code reasoning and data contamination. |
| Approach: | They instantiate dual-path reasoning in a benchmark and evaluate 13 LLMs. |
| Outcome: | The proposed model provides a robust and discriminative proxy for dynamic code understanding. |
Unlearn What You Want to Forget: Efficient Unlearning for LLMs (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) can be used to memorize a vast amount of data, but can suffer from privacy issues and data protection violations. |
| Approach: | They propose an efficient unlearning framework that could update LLMs without retraining them . they introduce lightweight unlearning layers learned with a selective teacher-student objective into transformers . |
| Outcome: | The proposed framework could update LLMs without having to retrain the whole model after data removals. |
Ontology-Guided Reverse Thinking Makes Large Language Models Stronger on Knowledge Graph Question Answering (2025.acl-long)
Copied to clipboard
| Challenge: | Existing methods rely on entity vector matching, but the purpose of the question is abstract and difficult to match with specific entities. Existing approaches rely only on entity-vector matching, and there is a problem with multi-hop reasoning. |
| Approach: | They propose a framework that constructs reasoning paths from purposes back to conditions using the KG ontology. |
| Outcome: | Experiments on the WebQSP and CWQ datasets show that ORT significantly improves the capability of large language models in knowledge graph question answering tasks (KGQA). |
Label and Explanation Variation in LLM-Based Annotation: a Case Study in Natural Language Inference (2026.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) have shown considerable promise for annotation purposes, but questions remain about their ability to capture human label variation (HLV) label variation is genuine disagreement between annotators observed across NLP tasks. |
| Approach: | They investigate how label and explanation variation manifests within and across LLMs with respect to the Natural Language Inference task. |
| Outcome: | The proposed models generate label distributions similar to humans but exhibit distinct, idiosyncratic judgments and disagreement patterns. |
Empowering Large Language Models for Textual Data Augmentation (2024.findings-acl)
Copied to clipboard
| Challenge: | True. True. False |
| Approach: | False slants are proposed to generate a large pool of augmentation instructions and select the most suitable task-informed instructions. |
| Outcome: | False omissions: the proposed approach consistently generates augmented data with better quality compared to non-LLM and LLM-based data augmentation methods. |
EpiCoDe: Boosting Model Performance Beyond Training with Extrapolation and Contrastive Decoding (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to enhance performance of Large language models are limited due to the cost of training data and privacy concerns. |
| Approach: | They propose a method that enhances a finetuned model with its inferior version and adopts contrastive decoding to reduce predicted errors. |
| Outcome: | The proposed method outperforms existing methods in data-scarcity scenarios across three domains and shows that it is more robust and robust. |
Human-Inspired Obfuscation for Model Unlearning: Local and Global Strategies with Hyperbolic Representations (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for unlearning large language models struggle to balance effective forgetting with maintaining model utility. |
| Approach: | They propose a human-inspired unlearning framework that simulates forgetting on fuzzy data and represents them in hyperbolic and Euclidean spaces. |
| Outcome: | The proposed framework is able to forget sensitive content while maintaining the model’s language understanding, fluency, and benchmark performance. |
KLAAD: Refining Attention Mechanisms to Reduce Societal Bias in Generative Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models exhibit societal biases in their outputs, prompting ethical and societal challenges. |
| Approach: | They propose an attention-based debiasing framework that implicitly aligns attention distributions between stereotypical and anti-stereotypical sentence pairs without directly modifying model weights. |
| Outcome: | The proposed framework improves on BBQ and BOLD benchmarks while maintaining fluency and coherence. |
Hopping Too Late: Exploring the Limitations of Large Language Models on Multi-Hop Queries (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) can solve complex multi-step problems, but little is known about how these computations are implemented internally. |
| Approach: | They propose a "back-patching" analysis method to solve multi-hop queries . they propose resolving the bridge entity into the bridge and the second hop into the target entity into latent steps. |
| Outcome: | The proposed method solves multi-hop queries that require two information extraction steps . it shows that the later layers lack the necessary knowledge to correctly generate the answer . |
VOYAGER: A Training Free Approach for Generating Diverse Datasets using LLMs (2026.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) are used to generate synthetic datasets but lack diversity . prior work has noted that such generated data lacks diversity - a problem that requires domain expertise. |
| Approach: | They propose a principled approach that optimizes a mathematical quantity that optimize the diversity of the dataset using determinantal point processes. |
| Outcome: | The proposed method improves diversity by 1.5-3 times compared to baseline approaches. |
Uniform Complexity for Text Generation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing models do not capture factors that contribute to producing consistent text. |
| Approach: | They propose a benchmark test to evaluate text complexity in generative models by observing linguistic properties of input prompts. |
| Outcome: | The proposed model fails to preserve complexity of input prompts even if finetuned with professionally written texts. |
DynaThink: Fast or Slow? A Dynamic Decision-Making Framework for Large Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) have emerged as prominent foundation models for diverse applications due to their outstanding ability to understand and generate humanlike text. |
| Approach: | They propose a dynamic decision-making framework that categorizes tasks into two distinct pathways: 'Fast' and 'Slow' they propose 'self-consistency' strategy to replace the straight-forward decoding method used in COT prompting . |
| Outcome: | The proposed method achieves more than 3% increase in accuracy with lower cost on five popular reasoning benchmarks. |
RetroLLM: Empowering Large Language Models to Retrieve Fine-grained Evidence within Generation (2025.acl-long)
Copied to clipboard
| Challenge: | Existing methods rely on separate retrievers to fetch top-k text chunks for generating evidence, and they lack joint optimization. |
| Approach: | They propose a framework that integrates retrieval and generation into a single, auto-regressive process, enabling LLMs to directly generate fine-grained evidence from the corpus with constrained decoding. |
| Outcome: | Extensive experiments on five open-domain QA datasets demonstrate the proposed framework’s superior performance across both in-domain and out-of-domain tasks. |
Enhanced Data Synthesis for LLM through Reasoning Structures Generated by Hierarchical GFlowNet (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to optimize instruction-response pairs lack a systematic design for the underlying reasoning structure. |
| Approach: | They propose a Reasoning Structure driven data Synthesis method that leverages a coarse-to-fine directed acyclic graph to construct reasoning structures efficiently. |
| Outcome: | The proposed method outperforms existing methods in 48.50%, 84.00%, 79.90% of the synthetic datasets trained on the proposed model. |
Structured Confidence–Guided Online Adaptation for LLM-based Multi-Label Classification (2026.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) enable zero-shot and few-shot multi-label text classification . but most approaches perform static inference and degrade under streaming test data . |
| Approach: | They propose a structured confidence-guided online adaptation framework for LLM-based multi-label generation without parameter updates. |
| Outcome: | The proposed framework improves Micro-F1 and Macro-F1, with the largest gains on long-tail labels. |
LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) are increasingly lengthy and require longer prompts . this paper presents a coarse-to-fine prompt compression method to reduce cost and increase performance. |
| Approach: | They propose a coarse-to-fine prompt compression method that maintains semantic integrity under high compression ratios and a token-level iterative compression algorithm to better model the interdependence between compressed contents. |
| Outcome: | The proposed method yields state-of-the-art performance and allows for up to 20x compression with little performance loss over four datasets from different scenarios. |
Not All Languages Are Created Equal in LLMs: Improving Multilingual Capability by Cross-Lingual-Thought Prompting (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) demonstrate impressive multilingual capability, but their performance varies substantially across different languages. |
| Approach: | They propose a generic template prompt that stimulates cross-lingual and logical reasoning skills to enhance task performance across languages. |
| Outcome: | The proposed method improves multilingual capability across languages and covers high-resource and low-resourced languages. |
TextGenSHAP: Scalable Post-Hoc Explanations in Text Generation with Long Documents (2024.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) are difficult to explain and understand due to long input contexts and autoregressive output generation. |
| Approach: | They propose a post-hoc explanation method which incorporates LLM-specific techniques. |
| Outcome: | The proposed method improves retrieval recall and prediction accuracy significantly on open-domain question answering benchmarks. |
JFLD: A Japanese Benchmark for Deductive Reasoning Based on Formal Logic (2024.lrec-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) have proficiently solved a broad range of tasks with their rich knowledge but struggle with logical reasoning. |
| Approach: | They propose a deductive reasoning benchmark for Japanese that assesses logical reasoning abilities isolated from knowledge and various reasoning rules. |
| Outcome: | The proposed benchmarks assess whether LLMs can generate logical steps to (dis)prove a given hypothesis based on a set of facts. |
TextLap: Customizing Language Models for Text-to-Layout Planning (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Creating 2D graphical layouts from text alone is challenging in traditional settings. |
| Approach: | They propose to customize LLMs to allow users to generate professional looking layouts by simply inputting text instructions. |
| Outcome: | The proposed method outperforms existing benchmarks for document generation and graphical design benchmarks. |
Enhancing Tool Learning in Large Language Models with Hierarchical Error Checklists (2025.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) have advanced natural language processing, but their effectiveness is often hampered by parameter mis-filling during tool calling. |
| Approach: | They propose a hierarchical tool error checklist framework to diagnose and mitigate tool-calling errors without relying on extensive real-world interactions. |
| Outcome: | The proposed framework improves parameter-filling accuracy and tool-calling success rates compared to baseline methods. |
Debate4MATH: Multi-Agent Debate for Fine-Grained Reasoning in Math (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing data annotation methods suffer from high annotation cost and lack of effective automatic validation. |
| Approach: | They propose a Fine-grained Multi-Agent Debate framework and a dataset that prompts multiple agents to debate and then a Multi-agent Debates Reward Model (MRM) to improve its mathematical reasoning capabilities. |
| Outcome: | The proposed model outperforms the state-of-the-art methods by 1.2% and 3.5% on a GSM8K dataset and 45.1% on the MATH dataset. |
From Reading to Compressing: Exploring the Multi-document Reader for Prompt Compression (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) have recently exhibited performance gains owing to a wide variety of prompting techniques, including Retrieval-Augmented Generation (RAG), Chain-of-Thought (CoT), and In-Context Learning (ICL). |
| Approach: | They propose a prompt compression method that captures the global context without compromising semantic consistency while detouring the necessity of pseudo-labels for training the compressor. |
| Outcome: | Empirical results show that the proposed method retains key contexts while reducing the prompt length by 80%. |
Text Fluoroscopy: Detecting LLM-Generated Text through Intrinsic Features (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) have revolutionized the field of natural language processing because of their excellent performance on various tasks. |
| Approach: | They propose a black-box method with better generalizability for detecting LLM-generated text by mining the intrinsic features of the text to be detected. |
| Outcome: | The proposed method achieves 7.36% and 2.84% improvement in detection performance compared to baselines in detecting texts from different domains generated by GPT-4 and Claude3 respectively. |
Large Language Models for Generative Recommendation: A Survey and Visionary Discussions (2024.lrec-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) have revolutionized the field of natural language processing but are not fully able to leverage the generative power of LLM. |
| Approach: | They examine the progress, methods, and future directions of large language models . they examine what generative recommendation is, why RS should advance to generative recommendations . |
| Outcome: | The proposed approach can be simplified to generate recommendations from the entire pool of items. |
FAME: Towards Factual Multi-Task Model Editing (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large language models embed extensive knowledge and perform exceptionally well across tasks. outdated knowledge or factual errors within LLMs can lead to misleading or incorrect responses. |
| Approach: | They propose to use a dataset to enhance the practicality of model editing to correct inaccurate information within LLMs. |
| Outcome: | The proposed method performs excellently across tasks and scenarios, confirming its practicality. |
Bridging Intuitive Associations and Deliberate Recall: Empowering LLM Personal Assistant with Graph-Structured Long-term Memory (2025.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs)-based personal assistants struggle to capture entity relationships and handle multiple intents effectively. |
| Approach: | They propose a graph-structured memory framework that mimics human cognitive processes and an event-centric memory graph. |
| Outcome: | The proposed framework outperforms retrieval and QA methods across long-term dialogue benchmarks and enables more human-like memory systems. |
The Instinctive Bias: Spurious Images lead to Illusion in MLLMs (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing multi-modal large language models (MLLMs) are able to process visual inputs by converting them into visual tokens that share the same latent space as language tokens in LLMs. |
| Approach: | They propose a benchmark that assesses the visual illusion level given spurious images and a pipeline that converts visual inputs into visual tokens. |
| Outcome: | The proposed benchmark shows that MLLMs suffer from an instinctive bias to varying degrees when presented with spurious images. |
VeriFact: Enhancing Long-Form Factuality Evaluation with Refined Fact Extraction and Reference Facts (2025.emnlp-main)
Copied to clipboard
| Challenge: | Prior work focuses on accuracy and precision, but factuality evaluation is difficult due to inter-sentence dependencies. |
| Approach: | They introduce a factuality evaluation framework to enhance fact extraction . they also introduce 'factRBench' that evaluates both precision and recall . |
| Outcome: | The proposed framework enhances fact extraction by identifying incomplete and missing facts . it also evaluates precision and recall in long-form models, whereas prior work focuses on precision. |
Distract Large Language Models for Automatic Jailbreak Attack (2024.emnlp-main)
Copied to clipboard
| Challenge: | Commercial large language models (LLMs) have made great progress in various NLP tasks. |
| Approach: | They propose a black-box jailbreak framework for automated red teaming of Large language models using an iterative optimization algorithm to conceal malicious content and memory reframing. |
| Outcome: | The proposed framework outperforms existing jailbreak defense methods and highlights the need to develop more effective and practical defense strategies. |
PersonaLens: A Benchmark for Personalization Evaluation in Conversational AI Assistants (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing personalization benchmarks focus on chit-chat, non-conversational tasks, or narrow domains, failing to capture complexities of personalized task-oriented assistance. |
| Approach: | They propose a benchmark to evaluate personalization in task-oriented AI assistants . the benchmark features user profiles equipped with rich preferences and interaction histories . |
| Outcome: | The proposed benchmark features user profiles equipped with rich preferences and interaction histories . it also features a judge agent and user agent that employs the LLM-as-a-Judge paradigm . |
LlamaCare: An Instruction Fine-Tuned Large Language Model for Clinical NLP (2024.lrec-main)
Copied to clipboard
| Challenge: | Large language models have shown remarkable abilities in generating natural texts . applying LLMs to clinical domain still poses significant challenges . |
| Approach: | They propose a method of instruction fine-tuning for adapting large language models to clinical domains . they generate instructions, inputs, and outputs covering a wide spectrum of clinical services . |
| Outcome: | The proposed method outperforms baseline LLMs on clinical tasks . it requires domain adaptation, task-specific learning, and reliability . |
Investigating Large Language Models for Complex Word Identification in Multilingual and Multidomain Setups (2024.emnlp-main)
Copied to clipboard
Răzvan-Alexandru Smădu, David-Gabriel Ion, Dumitru-Clementin Cercel, Florin Pop, Mihaela-Claudia Cercel
| Challenge: | Large language models (LLMs) are popular in the Natural Language Processing community because of their versatility and capability to solve unseen tasks in zero/few-shot settings. |
| Approach: | They investigate the use of large language models in CWI, LCP, and MWE settings by evaluating their use in zero-shot, few-shot and fine-tuning settings. |
| Outcome: | The proposed models struggle in certain conditions or achieve comparable results against existing methods. |
Continual Safety Alignment via Gradient-Based Sample Selection (2026.findings-acl)
Copied to clipboard
| Challenge: | Large language models require continuous adaptation to new domains, tasks, and evolving requirements. |
| Approach: | They propose a gradient-based sample selection method that filters high-gradient samples during fine-tuning. |
| Outcome: | The proposed method significantly improves alignment preservation while maintaining competitive task performance on continual domain tasks. |
Reversal of Thought: Enhancing Large Language Models with Preference-Guided Reverse Reasoning Warm-up (2025.acl-long)
Copied to clipboard
| Challenge: | Existing methods to improve LLMs’ logical capabilities involve traceable or verifiable logical sequences that generate more reliable responses yet increase computational costs, or introduce rigid logic template rules, reducing flexibility. |
| Approach: | They propose a plug-and-play reasoning framework that enhances LLMs' logical reasoning abilities during the warm-up phase prior to batch inference. |
| Outcome: | The proposed framework surpasses baselines in both reasoning accuracy and efficiency. |
Fill In The Gaps: Model Calibration and Generalization with Synthetic Data (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing calibration methods negatively impact model accuracy due to the lack of diversity of validation data. |
| Approach: | They propose a calibration method that incorporates synthetic data without compromising accuracy. |
| Outcome: | The proposed method improves model accuracy on real data and reduces calibration error by 34% on four different tasks. |
Make Prompt-based Black-Box Tuning Colorful: Boosting Model Generalization from Three Orthogonal Perspectives (2024.lrec-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) have shown increasing power on NLP tasks. however, tuning these models for downstream tasks usually requires exorbitant costs. |
| Approach: | They propose a black-box tuning technique that optimizes task-specific prompts without accessing gradients and hidden representations. |
| Outcome: | The proposed method improves performance under few-shot learning scenarios. |
Distributed LLM Serving on Consumer-Grade GPUs by Reconciling Computation and Communication (2025.findings-emnlp)
Copied to clipboard
Lewei Jin, Kui Zhang, Yongqi Chen, null Zhuoyifan, Renjie Li, Yi Gao, Bowei Yang, Zhengong Cai, Wei Dong
| Challenge: | Large language models are reshaping internet services, and serving them is costly. |
| Approach: | They propose an efficient distributed LLM serving system that splits prefill and decode requests into smaller chunks . |
| Outcome: | The proposed system reduces TTFT, TPOT, and latency compared to the state-of-the-art system. |
ExPerT: Personalizing LLM Responses to Users’ Domain Expertise via Query-Wise Semantic and Keystroke Behavioral Cues (2026.acl-long)
Copied to clipboard
| Challenge: | Existing personalization methods relying on static profiles or text-only signals fail to capture query-specific expertise variation. |
| Approach: | They propose a query-wise personalization framework that adapts LLM responses to query domain expertise by combining semantic and behavioral cues. |
| Outcome: | ExPerT reduces expertise inference error by 65.7% compared to the strongest baseline and improves response satisfaction by 17.52% on a 5-point Likert scale. |
Biology-Instructions: A Dataset and Benchmark for Multi-Omics Sequence Understanding Capability of Large Language Models (2025.findings-emnlp)
Copied to clipboard
Haonan He, Yuchen Ren, Yining Tang, Ziyang Xu, Junxian Li, Minghao Yang, Di Zhang, Yuan Dong, Tao Chen, Shufei Zhang, Yuqiang Li, Nanqing Dong, Wanli Ouyang, Dongzhan Zhou, Peng Ye
| Challenge: | Biology-Instructions is the first large-scale instruction-tuning dataset for multi-omics biological sequences. |
| Approach: | They propose a large-scale instruction-tuning dataset for multi-omics biological sequences . they propose 'chatMultiOmics' to overcome limitations of current LLMs on multi-ome tasks . |
| Outcome: | The proposed dataset bridges LLMs and complex biological sequence-related tasks while maintaining conversational fluency. |
Embedding Domain Knowledge for Large Language Models via Reinforcement Learning from Augmented Generation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to embed knowledge into large language models have some limitations . static nature of training data and lack of knowledge in domains create knowledge gaps . |
| Approach: | They propose a method that iteratively cycles between sampling generations and optimizing the model through calculated rewards. |
| Outcome: | The proposed method outperforms baseline approaches on medical, legal, astronomy, and current events datasets. |
AssistedDS: Benchmarking How External Domain Knowledge Assists LLMs in Automated Data Science (2025.findings-emnlp)
Copied to clipboard
An Luo, Xun Xian, Jin Du, Fangqiao Tian, Ganghua Wang, Ming Zhong, Shengchun Zhao, Xuan Bi, Zirui Liu, Jiawei Zhou, Jayanth Srinivasa, Ashish Kundu, Charles Fleming, Mingyi Hong, Jie Ding
| Challenge: | Large language models (LLMs) have advanced the automation of data science workflows, yet it remains unclear whether they can critically leverage external domain knowledge as human data scientists do in practice. |
| Approach: | They propose a benchmark to evaluate how large language models handle external domain knowledge in tabular prediction tasks. |
| Outcome: | The proposed model evaluates whether it can critically leverage external domain knowledge as human data scientists do in practice. |
Mixture of Soft Prompts for Controllable Data Generation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) generate fluent text when the target output follows natural language patterns. |
| Approach: | They propose a method that uses large language models to generate fluent text from a limited ontology rather than direct prediction by using soft prompts. |
| Outcome: | The proposed method produces diverse and natural text while preserving label semantics. |
Who Holds the Pen? Caricature and Perspective in LLM Retellings of History (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models are increasingly used to simulate human perspectives, authors say . authors: asymmetries in tone, stance, and emphasis can quietly, yet systematically, distort how history is told and remembered. |
| Approach: | They analyze LLM-generated responses across 197 historically significant events . they find that LLMs reliably distinguish persona-based responses from neutral baselines . |
| Outcome: | The findings show that LLMs distinguish persona-based responses from neutral baselines and that directly affected personas exhibit higher exaggeration. |
ParroT: Translating during Chat using Large Language Models tuned with Human Translation and Feedback (2023.findings-emnlp)
Copied to clipboard
Wenxiang Jiao, Jen-tse Huang, Wenxuan Wang, Zhiwei He, Tian Liang, Xing Wang, Shuming Shi, Zhaopeng Tu
| Challenge: | Large language models (LLMs) like ChatGPT are only accessible through restricted APIs, which creates barriers to new research and advancements in the field. |
| Approach: | They propose a framework to enhance and regulate the translation abilities during chat . they reformulate translation data into the instruction-following style and introduce a "Hint" field . |
| Outcome: | The proposed framework enhances and regulates the translation abilities during chat . it reformulates translation data into the instruction-following style and introduces a "Hint" field . |
Explain-then-Process: Using Grammar Prompting to Enhance Grammatical Acceptability Judgments (2025.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) can explain grammatical rules, but fail to apply those rules when judging sentence acceptability. |
| Approach: | They propose a grammar prompting paradigm that feeds an LLM's metalinguistic explanation back to the target model before deciding which sentence of a minimal pair is grammatical. |
| Outcome: | The proposed model improves on the English BLiMP, Chinese SLING, and Russian RuBLimp benchmarks. |
MotiveBench: How Far Are We From Human-Like Motivational Reasoning in Large Language Models? (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing benchmarks for large language models lack information asymmetry with real-world situations. |
| Approach: | They propose a benchmark to evaluate the human-like motivational and behavioral reasoning ability of LLMs with detailed, realistic situations. |
| Outcome: | The proposed benchmark compared LLMs with real-world scenarios on seven model families and found that the most advanced models struggle with understanding "love & belonging" needs. |
Assessing Privacy Risks in Language Models: A Case Study on Summarization Tasks (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models have revolutionized the field of NLP by achieving state-of-the-art performance on various tasks. |
| Approach: | They investigate the membership inference attack by using model's API to determine if a sample was part of the training data. |
| Outcome: | The proposed model is able to identify if a sample was part of the training data and exploits its similarity and resistance to document modifications as potential MI signals on widely used datasets. |
Length Representations in Large Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) have shown remarkable capabilities across various tasks, that are learned from massive amounts of text-based data. |
| Approach: | They propose to scale hidden units within the model to control output sequence length without losing the informativeness of the generated text. |
| Outcome: | The output sequence length is controlled by multiple head attention mechanisms, which can be adjusted in a disentangled manner. |
Semantic Inversion, Identical Replies: Revisiting Negation Blindness in Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Negation is a common occurrence in the real world and is essential for logical reasoning as it helps understand the opposite or absence of a statement. |
| Approach: | They propose a verification framework that includes task design and measurement methods to verify this phenomenon negation blindness on the query. |
| Outcome: | The proposed framework can be used to verify the model fails to capture semantic contradictions in negated queries despite its accurate understanding of knowledge about positive queries. |
Explicit Bayesian Inference to Uncover the Latent Themes of Large Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) have impressive generative capabilities, yet their inner mechanisms remain largely opaque. |
| Approach: | They propose a variational autoencoder-based neural topic model to interpret LLMs generation process through an explicit Bayesian framework by inferring latent topic variables via variational inference. |
| Outcome: | The proposed model outperforms state-of-the-art topic models on intrinsic measures of coherence and diversity on multiple datasets and shows significant gains on classification and summarization tasks. |
Which bird does not have wings: Negative-constrained KGQA with Schema-guided Semantic Matching and Self-directed Refinement (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing KGQA benchmarks and methods are biased toward positive and calculation constraints. Negative constraints are neglected, although they frequently appear in real-world questions. |
| Approach: | They propose a task where each question contains at least one negative constraint and a corresponding dataset, NestKGQA. |
| Outcome: | The proposed framework outperforms baselines on both KGQA and NEST-KGQA benchmarks under few-shot settings. |
Pause or Fabricate? Training Language Models for Grounded Reasoning (2026.findings-acl)
Copied to clipboard
Yiwen Qiu, Linjuan Wu, Yizhou Liu, Yuchen Yan, Jin Ma, Xu Tan, Yao Hu, Daoxin Zhang, Wenqi Zhang, Weiming Lu, Jun Xiao, Yongliang Shen
| Challenge: | Large language models implicitly fabricate information when inputs are incomplete, causing confidence but unreliable conclusions. |
| Approach: | They propose a framework for grounded reasoning under incomplete information that decomposes reasoning into two stages . they propose stage-specific rewards to penalize hallucinations, enabling models to detect gaps, stop proactively, and resume reasoning after clarification. |
| Outcome: | The proposed framework improves premise detection and task success by 30% . it also reduces average response length by over 20% . |
Bayesian Example Selection Improves In-Context Learning for Speech, Text and Visual Modalities (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) can adapt to new tasks easily and efficiently in a training-free manner. |
| Approach: | They propose to use eBayesian in-context example selection method to extend the inference probability conditioned on in-constitut examples based on Bayes’ theorem to select in-strategy examples . Experimental results show the efficacy and robustness of their method on various models, tasks and modalities. |
| Outcome: | The proposed method is based on the eBayesian in-context example selection approach. |
Building Trust in Clinical LLMs: Bias Analysis and Dataset Transparency (2025.emnlp-main)
Copied to clipboard
Svetlana Maslenkova, Clement Christophe, Marco AF Pimentel, Tathagata Raha, Muhammad Umar Salman, Ahmed Al Mahrooqi, Avani Gupta, Shadab Khan, Ronnie Rajan, Praveenkumar Kanithi
| Challenge: | Current dataset curation and bias assessment practices lack transparency . current approaches lack a thorough understanding of how data characteristics influence model behavior . |
| Approach: | They propose a comprehensive bias evaluation framework that integrates general benchmarks with a healthcare-specific methodology to probe for biases in a sensitive healthcare context. |
| Outcome: | The proposed approach to bias evaluation leverages established benchmarks and a healthcare-specific methodology. |
Mind the Style Gap: Meta-Evaluation of Style and Attribute Transfer Metrics (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) make it easy to rewrite a text in any style, but they are not straightforward when evaluating content preservation. |
| Approach: | They propose a large meta-evaluation of metrics for evaluating style and attribute transfer, focusing on content preservation. |
| Outcome: | The proposed method achieves higher alignment with human judgements than prompting a model of a similar size as an autorater. |
Do LLMs Align Human Values Regarding Social Biases? Judging and Explaining Social Biases with LLMs (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models can lead to undesired consequences when misaligned with human values . previous studies have shown misalignment of LLMs with human value using expert-designed or agent-based emulated bias scenarios . |
| Approach: | They investigate whether large language models (LLMs) are misaligned with human values . they find no significant differences in understanding of HVSB between LLMs . |
| Outcome: | The results show that large language models do not have lower misalignment rates and attack success rates . the study also shows that smaller language models have the ability to explain HVSB . |
Towards Fully Exploiting LLM Internal States to Enhance Knowledge Boundary Perception (2025.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) exhibit impressive performance across diverse tasks but struggle to accurately gauge their knowledge boundaries. |
| Approach: | They propose Consistency-based Confidence Calibration (C3) which assesses confidence consistency through question reformulation to improve LLMs’ ability to recognize their knowledge gaps. |
| Outcome: | The proposed method improves the unknown perception rate by 5.6% on NQ and 4.9% on HotpotQA. |
Beyond Similarity: A Gradient-based Graph Method for Instruction Tuning Data Selection (2025.acl-long)
Copied to clipboard
Yang Zhao, Li Du, Xiao Ding, Yangou Ouyang, Hepeng Wang, Kai Xiong, Jinglong Gao, Zhouhao Sun, Dongliang Xu, Qing Yang, Dongchen Li, Bing Qin, Ting Liu
| Challenge: | Existing methods for selecting training data from general datasets fail to account for the joint distribution of instructions, resulting in inefficient learning and suboptimal knowledge transfer. |
| Approach: | They propose a method that constructs a mixed gradient-based instruction graph to capture the joint distribution and interdependencies among instructions. |
| Outcome: | The proposed method outperforms existing methods on domain adaptation tasks and in complex, data-scarce scenarios. |
From Isolated Scoring to Collaborative Ranking: A Comparison-Native Framework for LLM-Based Paper Evaluation (2026.findings-acl)
Copied to clipboard
Pujun Zheng, Jiacheng Yao, Jinquan Zheng, Chenyang Gu, Guoxiu He, Jiawei Liu, Yong Huang, Tianrui Guo, Wei Lu
| Challenge: | Large language models (LLMs) are currently used to evaluate scientific papers by assigning an absolute score to each paper independently. |
| Approach: | They propose a comparison-native framework for paper evaluation that integrates comparison into both data construction and model learning. |
| Outcome: | The proposed framework achieves an average relative improvement of 21.8% over the strong baseline DeepReview-14B, while exhibiting robust generalization to five previously unseen datasets. |
Can Large Language Models Be Good Language Teachers? (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) have achieved remarkable success across diverse domains, but their potential as effective language teachers remains inadequately assessed. |
| Approach: | They propose a framework to evaluate Chinese language teachers' pedagogical competence against international standards. |
| Outcome: | The proposed framework evaluates 13 latest multilingual and Chinese LLMs against international standards for Chinese language teachers. |
DIESEL: A Lightweight Inference-Time Safety Enhancement for Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Large language models generate outputs that are not aligned with human values, such as toxic content, malicious use cases, and vulnerabilities to adversarial jailbreak attacks. |
| Approach: | They propose a lightweight inference-guidance technique that can be seamlessly integrated into any autoregressive LLM to semantically filter undesirable content during generation. |
| Outcome: | The proposed technique can be integrated into any autoregressive LLM to semantically filter undesirable content during generation. |
Toward Structured Knowledge Reasoning: Contrastive Retrieval-Augmented Generation on Experience (2025.findings-acl)
Copied to clipboard
Jiawei Gu, Ziting Xian, Yuanzhen Xie, Ye Liu, Enjie Liu, Ruichao Zhong, Mochi Gao, Yunzhi Tan, Bo Hu, Zang Li
| Challenge: | Large language models struggle to infer implicit relationships embedded in tabular formats . authors introduce a framework that builds experience memory representations and enhances generalization through contrastive In-Context Learning (ICL). |
| Approach: | They propose a framework that builds experience memory representations and enhances generalization through contrastive In-Context Learning to simulate human-like knowledge transfer. |
| Outcome: | Experiments on Text-to-SQL and TableQA show CoRE significantly improves performance . it achieves gains of 3.44% and 4.24%, with up to 17.2% on challenging tasks . |
FOLIO: Natural Language Reasoning with First-Order Logic (2024.emnlp-main)
Copied to clipboard
Simeng Han, Hailey Schoelkopf, Yilun Zhao, Zhenting Qi, Martin Riddell, Wenfei Zhou, James Coady, David Peng, Yujie Qiao, Luke Benson, Lucy Sun, Alexander Wardle-Solano, Hannah Szabó, Ekaterina Zubova, Matthew Burtell, Jonathan Fan, Yixin Liu, Brian Wong, Malcolm Sailor, Ansong Ni, Linyong Nan, Jungo Kasai, Tao Yu, Rui Zhang, Alexander Fabbri, Wojciech Kryscinski, Semih Yavuz, Ye Liu, Xi Lin, Shafiq Joty, Yingbo Zhou, Caiming Xiong, Rex Ying, Arman Cohan, Dragomir Radev
| Challenge: | Existing benchmarks for logical reasoning in large language models lack language naturalness or limited complexity. |
| Approach: | They propose to use first-order logic annotations to evaluate logical reasoning capabilities of large language models. |
| Outcome: | The proposed dataset evaluates the FOL reasoning ability of supervised fine-tuning on medium-sized language models. |
RevMUX: Data Multiplexing with Reversible Adapters for Efficient LLM Batch Inference (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) have brought a great breakthrough to the natural language processing community, but their high throughput demands make them difficult to handle concurrent queries. |
| Approach: | They propose a parameter-efficient data multiplexing framework that integrates a reversible design in the multiplexer and can be reused to perform reverse operations and restore individual samples for classification. |
| Outcome: | The proposed framework improves inference efficiency while maintaining satisfactory classification performance. |
KARPA: A Training-free Method of Adapting Knowledge Graph as References for Large Language Model’s Reasoning Path Aggregation (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for large language models (LLMs) are limited by step-by-step decision-making on KGs, or require fine-tuning or pre-training on specific KG. |
| Approach: | They propose a framework that harnesses the global planning abilities of large language models (LLMs) for efficient and accurate KG reasoning. |
| Outcome: | Extensive experiments show that the proposed framework achieves state-of-the-art performance in KGQA tasks, delivering both high efficiency and accuracy. |
Using tournaments to calculate AUROC for zero-shot classification with LLMs (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models perform well on zero-shot classification tasks, but are difficult to compare to supervised classifiers due to the lack of a modifiable decision boundary. |
| Approach: | They propose a method that transforms binary classification tasks into pairwise comparisons between instances within a dataset. |
| Outcome: | The proposed method produces relative rankings of instances using the Elo rating system . |
Small Models Struggle to Learn from Strong Reasoners (2025.findings-acl)
Copied to clipboard
Yuetai Li, Xiang Yue, Zhangchen Xu, Fengqing Jiang, Luyao Niu, Bill Yuchen Lin, Bhaskar Ramasubramanian, Radha Poovendran
| Challenge: | a small learning gap exists between large and small language models . long CoT data and large model responses are not beneficial for small models - a problem that may be due to the small student model's ability to handle distribution shifts. |
| Approach: | They propose a mix distillation strategy that balances reasoning complexity by combining long and short CoT examples or reasoning from both larger and smaller models. |
| Outcome: | The proposed strategy outperforms training on large and small models on short CoT and small model CoT. |
Improving Preference Alignment of LLM with Inference-Free Self-Refinement (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) develop in-context learning capability through pretraining and instruction tuning. |
| Approach: | Large language models (LLMs) develop in-context learning capability through pretraining and instruction tuning. |
| Outcome: | Experiments show that incorporating IFSR into preference alignment yields performance improvement over 10%. |
CLARO: Controlled Attribute-Driven Reasoning Optimization for Efficient Chain-of-Thought (2026.findings-acl)
Copied to clipboard
| Challenge: | Recent approaches guide reasoning length through token penalties or truncation, risking the omission of necessary steps. |
| Approach: | They propose a framework to internalize conciseness and attribute-driven reasoning optimization . they propose guiding models to embed high-quality structural attributes within a token budget . |
| Outcome: | The proposed method outperforms state-of-the-art models across benchmarks yielding accuracy gains of up to 63.6%. |
Uncertainty-Aware Contrastive Decoding (2025.findings-acl)
Copied to clipboard
| Challenge: | Large language models excel in a wide range of tasks, but generating factually accurate outputs remains a challenge. |
| Approach: | They propose a method that dynamically adjusts model contributions at each decoding step based on uncertainty. |
| Outcome: | The proposed method significantly improves factual accuracy and reliability over existing methods. |
SILO-BENCH: A Scalable Environment for Evaluating Distributed Coordination in Multi-Agent LLM Systems (2026.acl-long)
Copied to clipboard
Yuzhe Zhang, Feiran Liu, Yi Shan, Xinyi Huang, Xin Yang, Yueqi Zhu, Xuxin Cheng, Cao Liu, Ke Zeng, Terry Jingchen Zhang, Wenyuan Jiang
| Challenge: | Existing benchmarks conflate coordination ability with role-based priors. |
| Approach: | They propose a role-free benchmark for evaluating free-form collaboration under information silos. |
| Outcome: | The proposed benchmark systematically probes coordination capabilities under information silos using 54 configurations and 3 frontier LLMs. |
Does It Run and Is That Enough? Revisiting Text-to-Chart Generation with a Multi-Agent Approach (2025.findings-emnlp)
Copied to clipboard
| Challenge: | 15% of generated scripts fail to execute, even after supervised fine-tuning and reinforcement learning. |
| Approach: | They propose a lightweight multi-agent pipeline that separates drafting, execution, repair, and judgment . the system reduces execution errors to 4.5% within three repair iterations . |
| Outcome: | The proposed pipeline reduces execution errors to 4.5% within three repair iterations while requiring significantly less compute. |
Too Nice to Tell the Truth: Quantifying Agreeableness-Driven Sycophancy in Role-Playing Language Models (2026.acl-long)
Copied to clipboard
| Challenge: | Large language models are increasingly used as conversational agents that adopt personas and role-play characters at user request. |
| Approach: | They propose to examine how persona agreeableness influences sycophancy across 13 small, open-weight language models ranging from 0.6B to 20B parameters. |
| Outcome: | The proposed model consists of 275 personas and exposes them to 4,950 sycophancy-eliciting prompts spanning 33 topic categories. |
Tree-of-Debate: Multi-Persona Debate Trees Elicit Critical Thinking for Scientific Comparative Analysis (2025.acl-long)
Copied to clipboard
| Challenge: | Existing comparative summarization methods focus on surface-level semantic differences, which may not capture the most relevant distinctions. |
| Approach: | They propose a framework which transforms scientific papers into LLM personas that debate their respective novelties. |
| Outcome: | The proposed framework generates informative arguments and effectively contrasts papers, and supports researchers in their literature review. |
LinguaLens: Towards Interpreting Linguistic Mechanisms of Large Language Models via Sparse Auto-Encoder (2025.emnlp-main)
Copied to clipboard
| Challenge: | Prior research on linguistic mechanisms of large language models is limited by coarse granularity, limited analysis scale, and narrow focus. |
| Approach: | They propose a framework for analyzing the linguistic mechanisms of large language models based on Sparse Auto-Encoders. |
| Outcome: | The proposed framework extracts Chinese and English linguistic features across four dimensions . it uncovers intrinsic representations of linguistic knowledge in LLMs and can control outputs . |
Towards Generalizable and Faithful Logic Reasoning over Natural Language via Resolution Refutation (2024.lrec-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) have achieved significant performance in various natural language reasoning tasks, but struggle with performing first-order logic reasoning over formal logical theories expressed in natural language. |
| Approach: | They propose a framework which introduces the paradigm of resolution refutation to solve first-order logic reasoning problems by extending reasoning rules and employing the principle of proof by contradiction. |
| Outcome: | The proposed framework outperforms existing models while maintaining performance in simple scenarios. |
GFT: From Imitation to Reward Fine-Tuning with Unbiased Group Advantages and Dynamic Coefficient Rectification (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing studies have demonstrated that supervised fine-tuning and reinforcement learning are effective in integrating knowledge injection with robust generalization. |
| Approach: | They propose a unified post-training framework that addresses intrinsic limitations of supervised fine-tuning and reinforcement learning. |
| Outcome: | The proposed framework surpasses SFT-based methods and yields policies that integrate more smoothly with subsequent RL training. |
Fairness Evaluation and Inference Level Mitigation in LLMs (2026.findings-acl)
Copied to clipboard
| Challenge: | Large language models display undesirable behaviors embedded in their internal representations, undermining fairness, inconsistency drift, and the propagation of unwanted patterns during extended dialogues. |
| Approach: | They propose a pruning-based framework that detects context-aware neuron activations and applies adaptive masking to modulate their influence during generation. |
| Outcome: | The proposed framework detects context-aware neuron activations and applies adaptive masking to modulate their influence during generation. |
Large Language Models Threaten Language’s Epistemic and Communicative Foundations (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models are reshaping the norms of human communication, sometimes decouping words from genuine human thought. |
| Approach: | They propose to model humans, LLMs, and texts in a provenance network . they propose to use epistemic doppelgängers to generate texts that are indis- tinguishable from human-authored texts . |
| Outcome: | The proposed models induce semantic drift, erode account-ability, and obfuscate intent and authorship. |
EmoMAS: Emotion-Aware Multi-Agent System for High-Stakes Edge-Deployable Negotiation with Bayesian Orchestration (2026.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) are increasingly used for automated negotiation, but their cloud-centric paradigm exposes sensitive negotiations to privacy and security risks. |
| Approach: | They propose a Bayesian multi-agent framework that transforms emotional decision-making from reactive to strategic. |
| Outcome: | EmoMAS leverages a Bayesian orchestrator to coordinate three specialized agents: game-theoretic, reinforcement learning, and psychological coherence models. |
Optimizing User Profiles via Contextual Bandits for Retrieval-Augmented LLM Personalization (2026.acl-long)
Copied to clipboard
Linfeng Du, Ye Yuan, Zichen Zhao, Fuyuan Lyu, Emiliano Penaloza, Xiuying Chen, Zipeng Sun, Jikun Kang, Laurent Charlin, Xue Liu, Haolun Wu
| Challenge: | Existing approaches for personalizing large language models require modifying parameters. |
| Approach: | They propose a lightweight approach to personalizing large language models via retrieval augmentation . relevance serves as an unreliable proxy for utility, they argue . |
| Outcome: | The proposed framework outperforms strong heuristic and retrieval-augmented baselines on nine personalization tasks. |
Logit Space Constrained Fine-Tuning for Mitigating Hallucinations in LLM-Based Recommender Systems (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing LLM-based recommender systems rely on standard fine-tuning methodologies, often ignoring hallucination issues during the fine-uning process. |
| Approach: | They propose a logit space constraint-based fine-tuning framework to mitigate hallucination in LLM-based recommenders by incorporating Kullback–Leibler divergence into the training objective. |
| Outcome: | Experiments on two recommendation models with distinct LLM backbones and four real-world datasets show that LCFT reduces hallucination and enhances recommendation performance. |
Label Effects: Shared Heuristic Reliance in Trust Assessment by Humans and LLM-as-a-Judge (2026.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) are increasingly used as automated evaluators . et al., 2024: strong labels can foster trust but also undermine it . |
| Approach: | They show that LLMs' source labels bias trust judgments by humans . they use eye-tracking data to analyze LLM internal states during judgment . |
| Outcome: | The proposed model is biased by disclosed source labels, the authors show . eye-tracking data show humans rely heavily on source labels for judgments . |
Schoenfeld’s Anatomy of Mathematical Reasoning by Language Models (2026.acl-long)
Copied to clipboard
| Challenge: | Large language models expose reasoning traces, yet their underlying cognitive structure and steps remain difficult to identify and analyze beyond surface-level statistics. |
| Approach: | They propose a framework that explicitly abstracts reasoning traces into functional reasoning steps such as Analysis, Explore, Implement, Verify, etc. |
| Outcome: | The proposed framework reveals reproducible thinking dynamics and structural differences between reasoning and non-reasoning models, which are not apparent from token-level views. |
VET: Verifiable Execution Tracing for Reliable Text-to-SQL Generation (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for text-to-SQL generation are prone to hallucinations and grounding . authors present a novel reasoning paradigm that transforms text- to-Sql from unverifiable textual rationales into step-wise executable semantics. |
| Approach: | They propose a reasoning paradigm that transforms text-to-SQL from unverifiable textual rationales into step-wise executable semantics. |
| Outcome: | The proposed reasoning paradigm transforms text-to-SQL from unverifiable textual rationales into step-wise executable semantics. |
Improving Rule-based Reasoning in LLMs using Neurosymbolic Representations (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) face challenges in reliably solving reasoning tasks, especially when solving tasks that require strict rule following. |
| Approach: | They propose a method that encodes hidden states into neurosymbolic vectors and decodes them into a neurosample vector space to enable problem-solving within a neural space. |
| Outcome: | The proposed method shows an average of 88.6% lower cross-entropy loss and 15.4 times more problems correctly solved on a suite of mathematical reasoning tasks compared to chain-of-thought prompting and supervised fine-tuning (LoRA). |
Magnet: Multi-turn Tool-use Data Synthesis and Distillation via Graph Translation (2025.acl-long)
Copied to clipboard
Fan Yin, Zifeng Wang, I-Hung Hsu, Jun Yan, Ke Jiang, Yanfei Chen, Jindong Gu, Long Le, Kai-Wei Chang, Chen-Yu Lee, Hamid Palangi, Tomas Pfister
| Challenge: | Large language models have been shown to be effective in multi-turn interactions . however, their performance may be limited in complex, multi-turned interactions involving users and multiple tools. |
| Approach: | They propose a framework for synthesizing high-quality training trajectories to enhance the function calling capability of large language model agents in multi-turn conversations with humans. |
| Outcome: | The proposed model outperforms the teacher model by 68.01 on BFCL-v3 and 73.30 on ToolQuery. |
Training LLMs for Divide-and-Conquer Reasoning Elevates Test-Time Scalability (2026.acl-long)
Copied to clipboard
Xiao Liang, Zhong-Zhi Li, Zhenghao Lin, Eric Hanchen Jiang, Hengyuan Zhang, Yelong Shen, Kai-Wei Chang, Ying Nian Wu, Yeyun Gong, Weizhu Chen
| Challenge: | Large language models have demonstrated strong reasoning capabilities through step-by-step chain-of-thought (CoT) reasoning, but their strictly sequential nature constrains test-time scalability. |
| Approach: | They propose an end-to-end reinforcement learning framework to enhance LLMs' DAC-style reasoning capacity by decomposing a problem into subproblems and solving them sequentially. |
| Outcome: | The proposed model surpasses CoT by 8.6% and 6.3% on competition-level benchmarks and is available at the [github.com/MasterVito/DAC-RL]. |
Vocab Diet: Reshaping the Vocabulary of LLMs via Vector Arithmetic (2026.findings-acl)
Copied to clipboard
| Challenge: | Large language models often encode word-form variation as linear directions in the embedding space. |
| Approach: | They propose a compact reshaping of large language models' vocabulary by using shared vectors instead of unique tokens. |
| Outcome: | The proposed approach frees 10-40% of vocabulary slots to be reallocated where tokenization is inefficient. |
EVM-QuestBench: An Execution-Grounded Benchmark for Natural-Language Transaction Code Generation (2026.acl-long)
Copied to clipboard
| Challenge: | Existing evaluations of large language models overlook execution accuracy and safety. |
| Approach: | They propose an execution-grounded benchmark for natural-language transaction-script generation on EVM-compatible chains. |
| Outcome: | The proposed benchmark finds large performance gaps in the models with 5 independent rounds. |
Detecting LLM Hallucination Through Layer-wise Information Deficiency: Analysis of Ambiguous Prompts and Unanswerable Questions (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) often generate confident yet inaccurate responses, introducing significant risks for deployment in safety-critical domains. |
| Approach: | They propose a method to detect model hallucination by systematic analysis of information flow across model layers. |
| Outcome: | The proposed approach improves model reliability by immediately integrating with universal LLMs without additional training or architectural modifications. |
User Perceptions vs. Proxy LLM Judges: Privacy and Helpfulness in LLM Responses to Privacy-Sensitive Scenarios (2026.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) are rapidly being adopted for tasks like drafting emails, summarizing meetings, and answering health questions. |
| Approach: | They conducted a scenario-based evaluation of Large language models (LLMs) using 90 PrivacyLens scenarios. |
| Outcome: | The proposed models can leak private information in complex scenarios, but they do not measure user perceptions directly. |
ConSensus: Multi-Agent Collaboration for Multimodal Sensing (2026.findings-acl)
Copied to clipboard
| Challenge: | Large language models are increasingly grounded in sensor data to perceive and reason about human physiology and the physical world. |
| Approach: | They propose a training-free multi-agent collaboration framework that decomposes multimodal sensing tasks into specialized, modality-aware agents. |
| Outcome: | The proposed framework matches or exceeds debate methods on multimodal sensing benchmarks while achieving 12.7 times reduction in token cost. |
If an LLM Were a Character, Would It Know Its Own Story? Evaluating Lifelong Learning in LLMs (2026.acl-long)
Copied to clipboard
Siqi Fan, Xiusheng Huang, Yiqun Yao, Xuezhi Fang, Kang Liu, Peng Han, Shuo Shang, Aixin Sun, Yequan Wang
| Challenge: | Existing benchmarks for large language models (LLMs) fail to capture these dynamics, focusing on static, open-ended evaluations. |
| Approach: | They propose a benchmark to assess lifelong learning in large language models . they use two episodic datasets rich in narrative structure and character interactions . |
| Outcome: | Experiments on LLMs show that non-parametric methods outperform parametric ones in managing stateful learning. |
Query-Focused Individual Simulation with Progressive Persona Completion (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to simulating individual responses from persona information assume rich persona profiles, which are often unavailable in practice. |
| Approach: | They propose a query-focused individual simulation where relevant persona information is identified and requested on demand for each query. |
| Outcome: | Experiments on two dialogue datasets show that the proposed method achieves comparable performance to approaches that rely on rich persona information extracted from dialogue history. |
DIWALI - Diversity and Inclusivity aWare cuLture specific Items for India: Dataset and Assessment of LLMs for Cultural Text Adaptation in Indian Context (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing evaluation metrics for cultural awareness and alignment are lacking . Existing datasets for culture specific items (CSIs) focus primarily on concepts at the regional level and may contain false positives. |
| Approach: | They propose a new CSI dataset for Indian culture that measures cultural competence . they use a CSI created by LLM as Judge and human evaluations from diverse regions . |
| Outcome: | The proposed model shows that it is capable of generating culturally relevant adaptations across multiple cultural facets. |
Is Your Language Model Ready for Monetization Decisions? (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing benchmarks focus on shopping-centric scenarios and user-facing data, overlooking intermediate decision stages and robustness considerations. |
| Approach: | They propose a multi-task benchmark to evaluate large language models in real-world monetization contexts. |
| Outcome: | The proposed benchmark covers intent understanding, commercial matching, and user behavior modeling. |
Lookahead Q-Cache: Achieving More Consistent KV Cache Eviction via Pseudo Query (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing KV cache eviction methods prune tokens using prefilling-stage attention scores, causing inconsistency with actual inference queries. |
| Approach: | They propose a lookahead q-cache framework that generates low-cost pseudo lookaheaded queries to better approximate the true decoding-stage queries. |
| Outcome: | The proposed framework outperforms existing methods on LongBench and Needle-in-a-Haystack benchmarks and can be flexibly combined to yield further improvements. |
Efficient Real-time Refinement of Language Model Text Generation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) generate factually incorrect answers, a challenge that remains . Streaming-VR enables on-the-fly verification and correction of tokens as they are generated . |
| Approach: | They propose a method that enables on-the-fly verification and correction of LLM tokens as they are generated. |
| Outcome: | The proposed method improves factual accuracy and improves refinement efficiency compared to prior methods. |
Turning Logic Against Itself: Probing Model Defenses Through Contrastive Questions (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing safety measures detect overt malicious intent but fail to address subtle, reasoning-driven vulnerabilities. |
| Approach: | They propose a two-phase jailbreak technique that exploits contrastive reasoning to bypass safety mechanisms in large language models. |
| Outcome: | The proposed techniques achieve higher attack success rates (44%) than existing methods. |
A Survey of Toxicity Mitigation Strategies for Multilingual Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | Large language models can reproduce and amplify toxic content, including hate speech, harassment, and bias. |
| Approach: | They propose a comprehensive survey of the many detoxification methods tailored to multilingual LLMs. |
| Outcome: | The proposed methods are based on data filtering, style transfer, expert-based logit steering, retrieval augmentation, and human feedback. |
Activation Steering for Chain-of-Thought Compression (2026.findings-acl)
Copied to clipboard
| Challenge: | Large language models produce intermediate explanations, commonly referred to as chains of thought (CoTs), but the generated rationales are typically verbose, consuming many additional tokens, and thus degrading throughput and increasing inference energy consumption. |
| Approach: | They propose to generate concise reasoning traces by directly adjusting internal representations via activation steering. |
| Outcome: | The proposed method reduces generated token length by 69.4% across five reasoning benchmarks while maintaining accuracy. |
Robust Tool Use via Fission-GRPO: Learning to Recover from Execution Errors (2026.acl-long)
Copied to clipboard
Zhiwei Zhang, Fei Zhao, Rui Wang, Zezhong Wang, Bin Liang, Jiakang Wang, Yao Hu, Shaosheng Cao, Kam-Fai Wong
| Challenge: | Large language models (LLMs) can call tools effectively, but they remain brittle in multi-turn execution. |
| Approach: | They propose a framework that converts execution errors into on-policy corrective supervision within the RL training loop. |
| Outcome: | The proposed framework improves the error recovery rate of Qwen3-8B by 5.7% absolute and overall accuracy by 4.0% on BFCL v4 Multi-Turn. |
Lost in the Prompt Order: Revealing the Limitations of Causal Attention in Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | Large language models exhibit surprising sensitivity to structure of the prompt, but mechanisms underlying this sensitivity remain poorly understood. |
| Approach: | They conduct an in-depth investigation on placing context before the questions and options in MCQA prompts. |
| Outcome: | The proposed model outperforms the reverse order (QOC) by over 14%p over a wide range of models and datasets. |
Where Paths Split: Localized, Calibrated Control of Moral Reasoning in Large Language Models (2026.acl-long)
Copied to clipboard
| Challenge: | Large language models display heterogeneous moral preferences across settings. |
| Approach: | They propose a method for steering toward a desired ethical framework while preserving general competence. |
| Outcome: | The proposed method outperforms baselines while providing interpretable mechanism. |
LLM4Cell: Taxonomy and Evaluation of LLM and Agentic Models for Single-Cell Biology (2026.acl-long)
Copied to clipboard
Sajib Acharjee Dip, Adrika Zafor, Bikash Kumar Paul, Uddip Acharjee Shuvo, Muhit Islam Emon, Xuan Wang, Liqing Zhang
| Challenge: | Large language models are transforming biomedical discovery by linking molecular patterns with knowledge encoded in text. |
| Approach: | They propose to map 58 foundation and agentic models developed for single-cell research into eight key analytical tasks. |
| Outcome: | The proposed models are applied to eight key analytical tasks including annotation, trajectory inference, perturbation modeling, and drug-response prediction. |
Beyond the Leaderboard: Rethinking Medical Benchmarks for Large Language Models (2026.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) are proving significant potential in healthcare, prompting numerous benchmarks to evaluate their capabilities. |
| Approach: | They propose a framework that deconstructs benchmark development into five stages from design to governance and provides a checklist of 46 medically-tailored criteria. |
| Outcome: | The framework deconstructs benchmark development into five stages from design to governance and provides a comprehensive checklist of 46 medically-tailored criteria. |
Mechanistic Interpretability of Large-Scale Counting in LLMs through a System-2 Strategy (2026.findings-acl)
Copied to clipboard
Hosein Hasani, Mohammadali Banayeeanzade, Ali Nafisi, Sadegh Mohammadian, Fatemeh Askari, Mobin Bagherian, Amirmohammad Izadi, Mahdieh Soleymani Baghshah
| Challenge: | Large language models exhibit systematic limitations in counting tasks due to depth constraints. |
| Approach: | They propose a method that decomposes large counting tasks into smaller, independent sub-problems that the model can reliably solve. |
| Outcome: | The proposed method surpasses architectural limitations and achieves higher accuracy on large-scale counting tasks. |
ReTRE: Benchmarking LLM Transfer Robustness with Structure-Preserving Variants (2026.acl-long)
Copied to clipboard
| Challenge: | Learning transfer theory emphasizes that applying acquired knowledge to novel manifestations is a key signal of deep understanding |
| Approach: | They propose a benchmark that probes transfer robustness along two rewrite levels: Near Transfer and Far Transfer. |
| Outcome: | The proposed benchmark demonstrates that large language models are robust when faced with novel manifestations of the same problem. |
Learning Faster with Better Tokens: Parameter-Efficient Vocabulary Adaptation for Specialized Text Summarization (2026.acl-long)
Copied to clipboard
| Challenge: | a new approach to adapt generalist models to expert domains is needed to overcome this problem. |
| Approach: | They propose a parameter-efficient domain adaptation approach that combines vocabulary adaptation with pretraining for LLM-based text summarization. |
| Outcome: | The proposed approach reduces training time by 35-55% over continual pretraining and reduces parameter counts up to 37% w.r.t expansion-only methods. |
Efficient Hallucination Detection in Automatic Code Generation (2026.findings-acl)
Copied to clipboard
| Challenge: | Large language models produce source code that appears correct and well-formed, but includes hallucinated elements that cause downstream test failures. |
| Approach: | They develop a transformer-based detector that uses LLM internal representations to identify hallucinations. |
| Outcome: | The proposed detector outperforms existing methods and unsupervised methods in the code generation domain. |